Sensei's Library On Tour
Sensei's Library on Tour (SLoT) is a little Java tool, that converts a wiki dump into a set of HTML pages that can be easily converted for reading on a handheld computer or e-book reader.
This page deals with this tool and is maintained by Hans Wiezorke.
Feel free to add comment wherever you see fit.
I am looking for a volunteer to take over development. (See end of page)
History
One way to get SL onto a handheld, is to convert the SLSnapshot into a format for one of the many document readers out there. The problem with this approach is, that all the diagrams have to be scaled down, in order to fit onto a small screen. My first attempt to crunch the SLSnapshot with ISiloX resulted in a 40MB document, that only had badly scaled black and white diagrams.
The obvious solution for this is to rebuild the diagrams from their original ascii source and assemble them by putting together several small icons. The original idea for this approach came from AshleyF, who started PocketSL. Since many handheld computers do not come with a web browser, that can interpret JavaScript, I saw the need for a plain HTML version and thus wrote SLoT.
SLoT generates quite a lot of html (currently 62MB), but this html can be compressed very efficiently by web clipping applications. I use ISilo/ISiloX for compressing and viewing and the resulting document is 16MB. By filtering, you can cut this down to a few kB. Small subsets can even be viewed on older 2MB Palms.
Here's what you need:
- SLoT
- Arno kindly made it available here:
SLoT v0.50 (65KB)
- SL dump
- A
database dump (around 9MB download - 30MB unzipped) of the SL Wiki database (monthly updated).
(Note that your browser, e.g. Firefox, might automatically decompress the gz file. If so, just remove .gz from the filename and you're done.)
You do not need the SLSnaphot for this.
The readme file in the SLoT package contains all instructions needed.
SLoT features:
- Layout and navigation optimized for small screens
- A filtering mechanism, so you do not need to load the entire library
- A table of contents, that lists all pages
- Paths
- You can replace the tiny gifs, that make up the diagrams, with your own. If you make a nice set, please let me know.
- Works best on a high-resolution colour displays, but works fine on older black and white screens.
Future plans:
- Cleaning up the code (not really an example of good programming practice right now).
- Structuring the Table of contents. Right now there is one list of 6700 pages, that is not very user friendly. At least it's sorted. :-)
- Multi level item lists.
- Diagrams with coordinates.
- A version with ascii diagrams, that is easier to use on older 2MB Palms.
- Add a link to the diagrams, that displays SGF, that can be pasted into an SGF browser like PilotGOne.
This is how the SLoT output can be used with ISilo/ISiloX. ISilo is a shareware (30-days trial version) document browser for Palm and Pocket PC.
- Get iSilo (a document browser for handhelds) at
http://www.isilo.com.
- Get ISiloX (a document converter for isilo format) at
http://www.isiloX.com.
- Install iSilo on the handheld and iSiloX on my desktop.
- To convert an HTML document with iSilox, add it to your current project, enter the parameters (see below) and convert it. After that you should see the final iSilo document in your Install window in Palm Desktop. It is important to check the size and specify, that you want to install on the memory card, lest you exceed the amount of available RAM.
- Sync your Palm, this can take a while.
- Enjoy SLoT :-)
These Parameters worked for me:
- I chose html/toc.html as source document.
- Links: max depth 1, all three checkboxes unchecked.
- Images: no resizing, no improve contrast, no compression, 16-Bit
- Tables: include all table formatting.
- Color: ignore all bg color specified on body, keep text color
- Document: rigid, no default, home page 1
Discussion
MarkD: Nice work Hans, just tried it on my new Zire 71. Works great and looks very nice :)
There seems to be a small bug: if you run the program on a linux box, all generated files have names like "html\abcd.html".
HansWiezorke: Thanks. The \ is a bug, that works fine on Windows :). Easy fix: replace "html\\" with "html/" in Slot.java (line 63) and in myWiki/WikiWriter.java (line 47). then recompile Slot.java by typing:
javac Slot.java
This will be fixed in the next version.
MarkD: Thanks for the bugfix, figured that out myself tonight. Oh and: Nice new homepage for SLoT :)
Dronak: I'm a total newbie and would like to get just the basic files onto my PDA for now, but I don't know which ones those are. Does anyone already have a list of files that would be suitable for this purpose? A modified pagelist.txt file that I could run through SLoT would be good, but whatever you have would be appreciated. BTW, I'm doing the conversion with Plucker instead of iSilo and it seems to have problems going more than 1 link deep, so it would be helpful if the list didn't require going 2 links deep to get all of the information. If you can post a link to the file list, that would be great, but I can provide an e-mail address or something if needed. Thanks in advance.
Dronak: No replies yet, I see, but I have another question. Is it just me, my PDA, Plucker or something, or do boards wider than 13 points take two lines for everyone? It's hard to follow discussion using 19x19 board diagrams when each line takes up two and things get spread out vertically more than they should. The board just doesn't look right and I have a lot of trouble trying to correct for this display issue in my head. Just wondering if this is problem is local or global. Thanks.
Bug reports and wishlist for SLoT 0.50
- CamelCase links (with some uppercase character and no [ nor ]) are not taken into account. Same for escaped CamelCase strings (with !). In this case, the ! should be removed.
- Aliases links in path do not work: 'Page not found"
- As previously discussed, keyword filtering abiliy
- Lowercase letter-marks in diagrams (I can achieve that by myself, altering the small gifs)
- The character ' is escaped (\') in the page's titles and links and paths
- Some pages are quite useless when browsing (FindPage, AdvancedFindPage, RecentChanges, FullRecentChanges, RecentChangesGames, UserPreferences, MessagesToPeopleCurrentlyPresentInTheLibrary, SandBox...). They probably should be removed inconditionnally, or maybe be part of a black list of pages. Probably even all SLDescription tagged pages can be removed
- There is an extra space in the beginning of each diagram's caption
- Small bug: diagram title when coordinate mode is on might contain an extra 'c' in the beginning
- Allow
,
... notation
- Working footnotes
- Graphical frontend to select filtering options
- Remove external references/links for offline browsing
- Merging makepage and slot together along with filtering abilities
HansWiezorke: Hi Nico, thanks for the list.
- Most things you point out are already on my todo list.
- Altered Gifs: If you make a nice set of icons, let me know, and we can include it in the distribution.
I hope to get some stuff done this weekend. However I have a new very time consuming hobby, my son Benno (4 weeks old) :-).
MarkD: WOW! Congratulations Hans :)
Nico: I join MarkD, my best wishes to you.
Nico: I started to have a look at the code and try out a few things. Hans, you can expect some proposal patches soon...
Achieved so far (in a more or less crude way;):
- Corrected the / \ plateforme-dependant problem
- Rewrote the MakePageList with a TreeSet (let Java do the sorting)
- Filtered out Homepages and OngoingGames pagetypes
- Tokenized keywords + internal support for keyword filtering
- So, filtered out SLdescription pages as well
- Corrected the \' in page titles and paths and links
- Filtered out uneditable 'special pages' without keywords like UserPreferences...
- Removed extra spaces in diagram captions (thus correcting the extra 'c')
- Used the logging facility to correct some unbalanced links on SL, found a bug with external links in the beginning of lists
- Don't write a link in plain text followed by its name between () when its title and adress aren't the same. Put it in red as a normal. It is smaller, simpler and much more readable
- Add a glossary in the beginning of the main toc page. Also separated pages names per alphabetical blocks
- Add internal support for CamelCase recognition
HansWiezorke: That's great. It seems, you have accomplished a lot of stuff. The only thing I have gotten around to is adding style sheet support. Could you mail me a copy? We should put it all together and make a version 0.6.
Nico: Done ;) I am looking forward to the next version.
SergeStinckwich : I try it on my Linux box, but the links doesn't work, because they are all capitalized ... When the 0.6 version will be released ?
RobAnybody: I have taken the 0.5 code and copied and modified the writer class to output pocketsl-style XMLs (with several small modifications alowing in-text notations (,
). I've also fixed some minor bugs in the code (& not being converted to & ,
in diagrams, etc.). To whom shall I mail it? Or maybe I'll add for you guys finish with 0.6 and then merge my changes with the new code.
The resulting mass of files is quite smaller than original SLoT, but quite bigger than PocketSL (maybe the database dump has grown so much?) It is now 19M. I will try these days to modify makepagelist too see what will happen if I exclude homepages, humour pages etc.
HansWiezorke: For more than a year now I have not gotten around to putting together a version 0.6. I'll spare you all the lame excuses I have :-). So I am looking for a volunteer to take over development. I have several changes people mailed to me, that I can forward. Any volunteers please mail me (the address on my home page is still valid.
Pimenta: I see quite some interesting stuff has been made since 0.5, that still hasn't seen the light of day. I just bought a PDA today, and I'm quite interested in using SLoT, hence my interest. :-) The current version stops "compiling" the wiki db into html files halfway through (around letter t), and I cannot find out why.
Could you guys upload the "0.6" version to SL? Nico? Anybody? :-D Thanks.
Ciuli: Is the wiki dump still updated? I've tried downloading it but all I get is a short text file with some SQL statements...
Xemanu: I can't get the wiki dump with 9 Mb. Only a very short text file (648 kb). This file is available still and from where can be download?