Sensei's Library On Tour
|Table of contents
Sensei's Library on Tour (SLoT) is a little Java tool that converts a wiki dump into a set of HTML pages that can be easily converted for reading on a handheld computer or e-book reader.
If you have feature requests, bug reports, or questions about how to use SLoT, please use this page:
instead of (or in addition to) making a note here, where it may go unnoticed for some time.
No installation is needed, just put the uncompressed wiki-dump.sql file in the same folder as Slot-0.8.jar. You can then run Slot by typing:
java -jar Slot-0.8.jar
on the command-line, or just by double-clicking Slot-0.8.jar. Then sit back and watch the "html" folder fill up with beautiful Sensei's Library pages.
Major changes in this release include:
- GUI with progress bar
- restored toc.html generation
- optional homepage exclusion
- optionally use very small header/footer
- added HTML title tag to each page
- moved CSS to separate, linked file
- moved images to "i" folder to save space
To use in command-line mode (without the GUI), use this syntax:
- java -cp Slot-0.8.jar net.xmp.senseis.slot.Slot
Optional parameters are "--small" and "--nouserpages".
- Worked great for me! Thank you so much! June 2009, timan72
- This looks like a great tool - but unfortunately I couldn't get it to work :( Downloading the files and double-clicking on the .jar file brings up a dialog window - but when I click on "Start" it just hangs. Looking at processes in the task manager - javaw.exe is running but consuming no resources. I wondered if this might be due to my Java version - Java recently asked to upgrade itself, and I think the JRE is up to ?1.6.something - but returning to a previous version (1.4.2) did not help. This is on a windows XP machine. I'm sure the problem is with me - but I don't know how to go about solving it.AndrewG
- Download it here.
Major changes in this release include:
- Standalone jar packaging--just download and run Slot-0.7.jar and it will automatically create the html/ folder and put the generated files and images in it.
- Segregated files into separate directories, resulting in faster directory search times, e.g:
- Fixed references to subpages (e.g. Fuseki Exercise 1 / Attempts), and images now work properly on subpages as well.
- Great! Subpages and images work and because of the new folder structure the browsing is much faster. timan72
- Damnit! I thought it wasn't working properly, because it had only done until C och D or something, but it was working in the background... It would have been nice to know that (a progress bar is also always nice. How am I supposed to know when it ends? [I've got the task manager on now, so I can see when javaw.exe is no longer runing, so I know when it is done. Not an optimal solution]), as I had to wait for it to finish and do it all over again, since I started to delete everything in the html. Zarlan
- ...also why wasn't toc.html created. All the pages link to it (and I want it, so I can make an isilo file), but it doesn't seem to exist. Zarlan
- An easy and useful filter would be to omit all user homepages. (see http://senseis.xmp.net/?header=pagetype&term=HomePage)
- It would be nice if the pages could have a HTML-title tag with the page's name, because right now, if you place a bookmark, the default name is "Unknown" and you have to fill in something by hand.
- If it is not too difficult it would be nice if anything not absolutely necessary could be removed from the HTML-files in order to make them smaller (and reduce the rendering time in the PDA's browser). My ideas:
- omit the CSS-tag
- omit all the navigation and copyright text from the bottom of each page. (Instead a "html/index.html"-file could contain copyright information".)
- the images could be moved from <img src="../../images/nw.gif"> to <img src="../../i/nw.gif"> (This saves 5 bytes per intersection on the Go board and there are MANY intersections in SL)
Unzip Slot-0.6.zip, and put the uncompressed wiki-dump.sql file in the same folder as Slot.jar. You can then run the application by typing:
java -jar Slot.jar
on the command-line, or just by double-clicking Slot.jar if you're on OS X. Then sit back and watch the "html" folder fill up with beautiful Sensei's Library pages.
Please let me know if you have any problems, feature requests, difficulties, or ideas for how to improve SLoT. I want to make it as useful as possible for people, and that means I need as much feedback as possible! Thanks! --rosswerner
- That all main HTML-pages reside in one directory decreases the usefulness a lot because of two reasons: It makes many programs which prepare HTML for online readers fail and even in plain uncompressed HTML it makes changing pages on a PDA very slow, because the system has to search through the huge directory to find the right file. This could be helped if the directory could be splitted into alphabetic subdirectories (like "html/F/Fuseki.html"). Of course all links would have to be corrected.
- Links onto subpages do not work. An example can be found in "html/FusekiExercise1/Solution.html": The link is <a href="Solution.html"> where it should be <a href="FusekiExercise1/Solution.html">
- Board graphics and links on subpages do not work, because subpages reside in subdirectories and all links should therefore start with "../../whatever" but they start with only "../whatever". For an example look at "html/FusekiExercise1/Solution.html" in the slot's export. Workaround for graphics: Copy the images folder into the html folder. timan72
I'll try it with my Irex Illiad eBook reader (a really cool eInk device) MarkD
Cool, thanks! I'm interested to see if anyone can get the whole thing on a device as-is. I can generate all the HTML pages just fine on my desktop, but I tried to Plucker-ify them with no luck. (The various different builder programs I tried fail with various errors, perhaps because they can't handle so many pages?) --rosswerner
It works with the Illiad and looks really nice but it is extremly slow (several minutes to follow a link sometimes). Guess there are too many pages and images for the slow device. By the way, that's the Iliad: Irex Illiad MarkD
Thanks for the great tool! I ended up putting the thing without any compression on a 256 MB-SD-card. I also converted it to a 12 MB pdb with Sunrise ( http://sunrise.mobileread.com/), but browsing it with Vade Mecum, a Pocket PC pdb-reader, worked, but was slow.-- timan72
One way to get SL onto a handheld is to convert the SLSnapshot into a format for one of the many document readers out there. The problem with this approach is, that all the diagrams have to be scaled down, in order to fit onto a small screen. My first attempt to crunch the SLSnapshot with ISiloX resulted in a 40MB document, that only had badly scaled black and white diagrams.
SLoT generates quite a lot of html (currently 62MB), but this html can be compressed very efficiently by web clipping applications. I use ISilo/ISiloX for compressing and viewing and the resulting document is 16MB. By filtering, you can cut this down to a few kB. Small subsets can even be viewed on older 2MB Palms.
- Arno kindly made it available here: SLoT v0.50 (65KB)
- SL dump
- A database dump (around 9MB download - 30MB unzipped) of the SL Wiki database (monthly updated).
(Note that your browser, e.g. Firefox, might automatically decompress the gz file. If so, just remove .gz from the filename and you're done.)
You do not need the SLSnaphot for this.
The readme file in the SLoT package contains all instructions needed.
- Layout and navigation optimized for small screens
- A filtering mechanism, so you do not need to load the entire library
- A table of contents, that lists all pages
- You can replace the tiny gifs, that make up the diagrams, with your own. If you make a nice set, please let me know.
- Works best on a high-resolution colour displays, but works fine on older black and white screens.
- Cleaning up the code (not really an example of good programming practice right now).
- Structuring the Table of contents. Right now there is one list of 6700 pages, that is not very user friendly. At least it's sorted. :-)
- Multi level item lists.
- Diagrams with coordinates.
- A version with ascii diagrams, that is easier to use on older 2MB Palms.
- Add a link to the diagrams, that displays SGF, that can be pasted into an SGF browser like PilotGOne.
This is how the SLoT output can be used with ISilo/ISiloX. ISilo is a shareware (30-days trial version) document browser for Palm and Pocket PC.
- Get iSilo (a document browser for handhelds) at http://www.isilo.com.
- Get ISiloX (a document converter for isilo format) at http://www.isiloX.com.
- Install iSilo on the handheld and iSiloX on my desktop.
- To convert an HTML document with iSilox, add it to your current project, enter the parameters (see below) and convert it. After that you should see the final iSilo document in your Install window in Palm Desktop. It is important to check the size and specify, that you want to install on the memory card, lest you exceed the amount of available RAM.
- Sync your Palm, this can take a while.
- Enjoy SLoT :-)
These Parameters worked for me:
- I chose html/toc.html as source document.
- Links: max depth 1, all three checkboxes unchecked.
- Images: no resizing¹, no improve contrast, no compression, 16-Bit
- Tables: include all table formatting.
- Color: ignore all bg color specified on body, keep text color
- Document: rigid, no default, home page 1
1. Pallando: Depending on your device, it may be necessary to adjust these settings in order to view whole board positions correctly. See my post below.
MarkD: Nice work Hans, just tried it on my new Zire 71. Works great and looks very nice :)
There seems to be a small bug: if you run the program on a linux box, all generated files have names like "html\abcd.html".
HansWiezorke: Thanks. The \ is a bug, that works fine on Windows :). Easy fix: replace "html\\" with "html/" in Slot.java (line 63) and in myWiki/WikiWriter.java (line 47). then recompile Slot.java by typing:
This will be fixed in the next version.
MarkD: Thanks for the bugfix, figured that out myself tonight. Oh and: Nice new homepage for SLoT :)
Dronak: I'm a total newbie and would like to get just the basic files onto my PDA for now, but I don't know which ones those are. Does anyone already have a list of files that would be suitable for this purpose? A modified pagelist.txt file that I could run through SLoT would be good, but whatever you have would be appreciated. BTW, I'm doing the conversion with Plucker instead of iSilo and it seems to have problems going more than 1 link deep, so it would be helpful if the list didn't require going 2 links deep to get all of the information. If you can post a link to the file list, that would be great, but I can provide an e-mail address or something if needed. Thanks in advance.
Dronak: No replies yet, I see, but I have another question. Is it just me, my PDA, Plucker or something, or do boards wider than 13 points take two lines for everyone? It's hard to follow discussion using 19x19 board diagrams when each line takes up two and things get spread out vertically more than they should. The board just doesn't look right and I have a lot of trouble trying to correct for this display issue in my head. Just wondering if this is problem is local or global. Thanks.
Problems with viewing whole-board positions
Problem #1: The issue appears to depend on the resolution of the handheld. I own a Palm Zire 31 with 160x160 Pixels, for instance, which is too small to display larger board positions without adjusting the picture size.
Solution #1: Each point on the board is a separate image. In order to shrink these pictures, you have to choose very small picture sizes; on my Zire 31, I had to choose a size of 7x7 pixels per image to make whole-board positions fit.
First, figure out your screen resolution (check the manual or the web for details). Then, divide the width by 19 and see what turns up.
Problem #2: However, by doing that all diagrams will shrink in size, making small diagrams such as corner josekis harder to read.
Solution #2: -?- I cannot think of anything except for either keeping two copies of Sensei's on your card and switching between the two, or decide whether to put your focus on whole board or corner positions. I'd be glad for any suggestions concerning that! - Pallando (04-02-06)
- CamelCase links (with some uppercase character and no [ nor ]) are not taken into account. Same for escaped CamelCase strings (with !). In this case, the ! should be removed.
- Aliases links in path do not work: 'Page not found"
- As previously discussed, keyword filtering abiliy
- Lowercase letter-marks in diagrams (I can achieve that by myself, altering the small gifs)
- The character ' is escaped (\') in the page's titles and links and paths
- Some pages are quite useless when browsing (FindPage, AdvancedFindPage, RecentChanges, FullRecentChanges, RecentChangesGames, UserPreferences, MessagesToPeopleCurrentlyPresentInTheLibrary, SandBox...). They probably should be removed unconditionally, or maybe be part of a black list of pages. Probably even all SLDescription tagged pages can be removed
- There is an extra space in the beginning of each diagram's caption
- Small bug: diagram title when coordinate mode is on might contain an extra 'c' in the beginning
- Allow ,... notation
- Working footnotes
- Graphical frontend to select filtering options
- Remove external references/links for offline browsing
- Merging makepage and slot together along with filtering abilities
HansWiezorke: Hi Nico, thanks for the list.
- Most things you point out are already on my todo list.
- Altered Gifs: If you make a nice set of icons, let me know, and we can include it in the distribution.
I hope to get some stuff done this weekend. However I have a new very time consuming hobby, my son Benno (4 weeks old) :-).
MarkD: WOW! Congratulations Hans :)
Nico: I join MarkD, my best wishes to you.
Nico: I started to have a look at the code and try out a few things. Hans, you can expect some proposal patches soon...
Achieved so far (in a more or less crude way;):
- Corrected the / \ plateforme-dependant problem
- Rewrote the MakePageList with a TreeSet (let Java do the sorting)
- Filtered out Homepages and OngoingGames pagetypes
- Tokenized keywords + internal support for keyword filtering
- So, filtered out SLdescription pages as well
- Corrected the \' in page titles and paths and links
- Filtered out uneditable 'special pages' without keywords like UserPreferences...
- Removed extra spaces in diagram captions (thus correcting the extra 'c')
- Used the logging facility to correct some unbalanced links on SL, found a bug with external links in the beginning of lists
- Don't write a link in plain text followed by its name between () when its title and address aren't the same. Put it in red as a normal. It is smaller, simpler and much more readable
- Add a glossary in the beginning of the main toc page. Also separated pages names per alphabetical blocks
- Add internal support for CamelCase recognition
HansWiezorke: That's great. It seems, you have accomplished a lot of stuff. The only thing I have gotten around to is adding style sheet support. Could you mail me a copy? We should put it all together and make a version 0.6.
Nico: Done ;) I am looking forward to the next version.
SergeStinckwich : I try it on my Linux box, but the links doesn't work, because they are all capitalized ... When the 0.6 version will be released ?
RobAnybody: I have taken the 0.5 code and copied and modified the writer class to output pocketsl-style XMLs (with several small modifications allowing in-text notations (, ). I've also fixed some minor bugs in the code (& not being converted to & , in diagrams, etc.). To whom shall I mail it? Or maybe I'll add for you guys finish with 0.6 and then merge my changes with the new code.
The resulting mass of files is quite smaller than original SLoT, but quite bigger than PocketSL (maybe the database dump has grown so much?) It is now 19M. I will try these days to modify makepagelist too see what will happen if I exclude homepages, humour pages etc.
HansWiezorke: For more than a year now I have not gotten around to putting together a version 0.6. I'll spare you all the lame excuses I have :-). So I am looking for a volunteer to take over development. I have several changes people mailed to me, that I can forward. Any volunteers please mail me (the address on my home page is still valid.
Pimenta: I see quite some interesting stuff has been made since 0.5, that still hasn't seen the light of day. I just bought a PDA today, and I'm quite interested in using SLoT, hence my interest. :-) The current version stops "compiling" the wiki db into html files halfway through (around letter t), and I cannot find out why.
Could you guys upload the "0.6" version to SL? Nico? Anybody? :-D Thanks.
usagi: With respect to what brianlee said below and what P. said above, I can confirm that SLoT does not produce a complete series of HTML files and is therefore unusable. It stops around letter t for no apparent reason. System: Linux
Ciuli: Is the wiki dump still updated? I've tried downloading it but all I get is a short text file with some SQL statements...
Xemanu: I can't get the wiki dump with 9 Mb. Only a very short text file (648 kb). This file is available still and from where can be download?
brianlee: Has anyone managed to generate the HMTL SLoT output using the latest SL Dump? I can successfully create the PageList.txt but when I try to generate the HMTL, my PC just hangs for a long time (1hr+)...
Pallando: I just wanted to add that I had to manually create a folder "html" for the "java Slot"-command to work. (I'm using Windows XP...)
Jano: When I was creating the html pages with "java Slot" I got an OutOfMemoryError and same with "java -mx80m Slot". With "java -mx180m Slot" everything worked nicely (Windows XP too ... )
Alejo The number you put is the MB of extra RAM that you give to the java program to process the file. The more you give, the faster it'll be made, but be carefull no to give too much memory, your system would crash and startup again...