SGF Organizer

This is the homepage of the SGF Organizer Project

Please feel free to add you own ideas, comments etc. to the marked (*) sections. Click on a blue section number to jump to that part of the page.

I am looking for some help...please check out the Requests section.

Preface

SGF Organizer is a small and simple tool currently under development by me. I am looking for ideas, suggestions, etc.
It will be released as freeware, so no commercial advertisment here.
Contributions (money, beer, used go books,...) are always welcome of course.

SGF Organizer helps you organize all the SGF files on your harddisk. Maybe you know the problem: You have downloaded hundreds or even thousands of SGF records and many of them have names like "0043334.sgf" etc. It's nearly impossible to find a game record after some time.

One solution is to rename every file after downloading it. But if you download a zip archive with many files (sometimes hundreds of games in one archive) that takes a lot of time.

That's why I am creating SGF Organizer.
In the first version, there will be (or...should be :) ) the following functions:

• Scan a directory or a whole tree of directories (or your entire harddisk) for SGF game records
• Store them all in a searchable database
• Display additional informations about the SGF file (player names and ranks, date and place of the game etc.). The SGF format allows you to enter a lot of information about a game.

Current Project Status

Completed:

• Scanning directory trees, save and load databases
• Scan SGF files, extract and show details
• Advanced search functions (using several fields with "and" or "or"). Some examples: "Show me all games of 9 dan players", "...of dan against kyu players", "...where white was 9 dan and lost the game", "...of player O Rissei which he lost",...
• Categories (GUI/functions)
• Full text search
• Basic GUI collections, basic functions for collections
• Drag & Drop from/to collections
• Basic renaming functions, I will post a screenshot soon
• Copy a Collection to a directory, delete files from disc
• Find duplicates in a collection

In progress:

• bug search, bug search, bug search, bug search, ...
• More collection functions...(suggestions?)
  • AshleyF: Can a file be in more than one collection/category at a time? That would be nice.
    MarkD: A file has exactly one category, but it can be in an unlimited number of collections.

Next:

• Custom field order
• User preferences screen
• Edit mode to alter SGF properties
• bug search, bug search, bug search, bug search,... :)

Planned features

• Palm Pilot export. Allows export to Palm PDAs without the 4k file size limitation.
• Light weight web server. Allows users to connect and view / download your game records over the internet.
• Organize game records in "Categories" (e.g. Pro Games, Own Games, Title Matches,...)
• Organize game records in "Collections"...you can add records to a collection and export these files to another directory or your Palm (see below). This feature allows you to add files from different categories. Examples for a collection: Reviewed Games, Very interesting games, Laptop Export, ....
• Renaming engine: Rename your SGF files using variables extracted from the files. Example: Rename all files using player name white + player name black + date, rename all files using date + event + round, etc.
• Support for multiple database files. Because SGF Organizer uses a flat file database system, it's possible to create a CD ROM with game records and a database.
• Exporting of game lists to a text or Microsoft Excel file

Screenshots

Screenshot of the main window. At the left: Collections & Categories tree view, at the bottom: message window for system messages


Screenshot of the detail window.


Screenshot of the search page.


Screenshot of the renaming engine, very easy to use.


Screenshot of the duplicates search page.

Download

Not yet, please be patient.

Technical Notes

• Code developed using Borland C++ Builder 4.0 Prof.
• I am not using the BDE (Borland Database Engine). It is several MB big and has problems with some installed programs on some systems. SGF Organizer uses mylittlebase, a very simple open source database. It adds only a few kb to the final code size.
• Speed: It takes about 2 minutes to scan a 40 GB harddisc partition with over 6400 SGF files on it. I guess some of the algorithms could be faster, but the biggest speed limitation is the hard disk or CD ROM read process. Try to search for a file with the windows explorer search function, that takes some time. SGF Organizer obviously can't be faster than that, because it uses the windows read functions. (for comparison: I am using an AMD 2400 system with 512 MB RAM and Windows NT4.0)
• Easy to install...no need for an installer, everthing you need is in one directory
• The SGF file format provides the following fields: (Note: Not every field has to be used, some may remain empty in some SGF files)
  • Name: name of the game record
  • Size: size of the board
  • Komi: komi
  • White Name: name of the white player
  • White Rank: rank of the white player
  • Black Name: name of the black player
  • Black Rank: rank of the black player
  • White Team: name of the white team
  • Black Team: name of the black team
  • Result: result
  • Date: date of the game
  • Event: name of the event (e.g. 58th Honinbo Title Match)
  • Place: place where the game was played (e.g.: Tokyo)
  • Round: round (e.g.: 5th game)
  • Rules: ruleset (e.g.: japanese)
  • Time limit: time limit in seconds
  • Opening: information about the opening (fuseki), rarely used in any file
  • Overtime: overtime system
  • Annotations: name of the person commenting the game
  • User: name of the person who created the SGF file
  • Copyright: copyright information
  • Source: source of the file
  • Application: application that was used to create the file (e.g. CGOban2,...)

Known Bugs and Limitations

• Current version only supports windows
• Available only in English...I may translate it later, but the program is easy to understand, maybe it's not necessary to translate it
• No unicode support for Japanese etc...but see note above
• No help system...see second note
• No support for multiple games in one SGF file (like most SGF tools)
• Very few error handlers yet...I will add more while testing the program.
• Search engine limited to one field of each type per query. (see screenshot of the search page)

Suggestions

Please feel free to add your own suggestions here:

• Tagging and searching system like Adobe Photoshop Album
  • This allows the user to create a hierarchy of 'Tags' representing their taxonomy, then assign multiple tags to items. User can then pivot on any set of points in the taxonomy to do searches. Very, very cool. (Example: http://www.feniello.com/go/album.png) -- Thanks Ashley, looks like a nice sorting system.
  • A rudimentary form of what I was suggesting is built into Go Album. You could do a much better job with the UI having more screen real estate. After a few months of playing with it though, I'm convinced that tagging (or some other metaphore for it) is the coolest way to organize problems, games, etc. Your idea of collections and categories may be the same (if SGFs can simultaneously exist in multiple). The idea of stacks in WinFS is also pretty much the same thing. It'll be great to have a good organizer. Any word on a release date for SGFOrganizer?

• A flexible script file renaming routine, with variables to add in the filenames, like Name_White, Name_Black, Name_Event, Index_Number_Of_Game_Record_In_Currently_Open_Collection, et cetera. The more the merrier.^[81]

• Stefan: Finding and eliminating duplicate game records. Useful for the more disorganised among your audience (ahum). Also this looks like a useful thing to have if people will start merging their databases. Thanks for your efforts, I already look forward to the day your craft will fill this gap in go software.
  MarkD: SGF Organizer already calculates a unique checksum for every game record. I still have to program the user interface for finding duplicates. Sometimes keeping duplicates may be useful, for example if you are collecting every game for Player A and Player B in certain directories. If Player A has ever played Player B, there is a SGF file in each directory. This may be intended or not.
  AshleyF: I would say that if SGF Organizer is good enough, I would cease to care about how files are arranged on disk. There should be an 'Export Search Set' feature to gather SGFs by player name, etc. and export to a set of files on disk to take on your Pocket PC or some such purpose. I don't see any reason to organize the files on disk and also organize them in the tool.
  MarkD: The export to directory function is already on my list. I am currently doing the GUI for "collections" and "categories". The results of the last search are always in one collection (see section screenshots ^[4], first picture, left part). There will be a function "copy collection to directory".

• Scryer: Optionally replace names using Asian characters with Roman characters. Same process with names of tournaments.

• Sebastian: Find modified versions of the same game. E.g., a file is saved each time I get disconnected on IGS. SGF Organizer should inform me that x-y.sgf is the predecessor of x-y_1.sgf. I'm afraid that a checksum can not find this. /030909/

• Sebastian: Merge versions into one file. /030909/

• Sebastian: Export to Pocket PC. /030909/

• Sebastian: Import (and merge) from Pocket PC. /030909/

• Sebastian: Display Game Comment (GC field). /030909/

8.1 Variables:

• Sebastian: Customizable columns, using variables as defined below^[81] /030909/

• • Sebastian: Where they correspond to SGF properties, use SGF standard abbreviations for the variable names, like PW, PB, EV. This should be implemented in a very simple way. The program doesn't need to know if XX is really an SGF property or not. If a user enters XX in a formula, then just return the first such attribute (if it exists). /030909/

• • Sebastian: "Number Of" operator - counts attributes: #AW would mean: Number of white stones in root node. (I think you don't need to worry about compressed ranges.) /030909/

• • Sebastian: Simple Arithmetic and logical operations, such as {if #AB=HA&#AW=0;Game;Problem} /030909/

• • Sebastian: regular expressions

• losdog: I would like to be able to open an SGF from a list using a choice of editors... for example, when you right click on a file name a context menu pops up -> "open with...Kombilo, glGo, etc."

• willemien: At the moment I am studying go on small boards. And would like to integrate all my games on boards of the same size in one SGF file. (so i end op with one 5x5, 7x7 and 9x9 games file.) So that when I study I can see the variants on one goban. (and add a lot of other variants to them in the same time) Am not sure about all the technicalities yet. I guess it is also would be useful for joseki study. (instead of opening another joseki file all similar joseki's are just merged together in the same board/sgf tree)

Bug Reports

Please add bugreports here.

• Username - Date of entry - Bug description
• ...

Discussion

Thank you all very much for your input and your help.

MarkD: The directory scan functions are already working (speed: It takes about 2 minutes to scan a 40 GB harddisk partition with 6400 SGF files on it. It may be possible to improve that. But thats not on top of my list, because I think that's fast enough. You don't scan your entire disk every day, but normally only once. After downloading a new file or archive you only have to scan the new directory).

Hu: Here is how I organize my records: The file system hierarchy has directories for names I have played under (Hu), Classic Games, Complete Games (recent) by other players, specific directories for particular players of interest like Takemiya and Go Seigen, Joseki, Problems, Student for when I've received a teaching game, and Teacher for when I've given a teaching game or reviewed other players' games. Under the Hu directory, I have sub-directories for the sizes I played (7x, 9x, 11x, 13x, 15x, 17x, 19x, 21x) and each of those has subdirectories for Won and Lost. Each game is saved with a filename according to this scheme: BlackName-WhiteName-RH3D-2003MMDD.sgf, for a rated 3 handicap game with discussion afterward, played in the MM month on DD day (F for free, S for simultaneous, T for teaching).

Of course, a proper database would make for much better organization and retrieval. Search terms should include player names, player ranks, ruleset, board size (include non-square), handicap, komi, time regime, time settings, result, result margin (if applicable), number of moves, date, where played. Other possibilities are game name, event, round, source, copyright (?!), annotator, transcriber. It would be good to be able to search for words in the commentary, for example to get all the games a particular observer may have commented in. Another idea would be to create entries for games not on local disk, but elsewhere such as the internet. -- Hu.

MarkD: Hu, do you have an example of a non square board SGF file? I have only SGF files with one number (mostly 9,13,19 of course) for square boards.

Velobici: MarkD, this sounds like a great project. 2 minutes to scan all the files is fine. It can be done out of cron every night while the user is sleeping....speed is not very important here. Will it work on Unix as well? Does the Borland developement environment you are using support The X Window system?

MarkD: Borland C++ Builder only supports Windows as far as I know.

Velobici: Ah..that is a shame. Any chance of interesting you in Java for this project. That would support Windows, Mac, Unix and cell phones.

MarkD: I am not really good at Java...but I know someone who is *looking at Hans Wiezorke*

HansWiezorke: Well, Java has indeed its merits, when it comes to portability and stability. I could help you with your first steps with Java, if you like. If you are fluent in C++, it should be no big deal. The only thing that takes a little getting used to is the GUI libraries. Drop me a mail, if you are interested.

MarkD: Hans, I have done some projects in Java, but only small programs with no GUI. There is already some work done for the SGF Organizer C++ version, so I don't really want to change the programming language now.

MarkD: 31/07/03 - Major page update. Master edit of the old page: I've moved the discussion to an own section, cleaned up some stuff and added a menu at the top of the page. I hope you like the new layout :)

DudeG: Well, here's another suggestion for multiple platform development-- Qt. It's still the basic C++ so that wouldn't require any changes, the only thing would be learning the Qt graphics calls. Qt is free for use just so long as the programs dveloped for it are also free for use (basically--you should read the license for full details, because I forget the specifics). Anyway, I really hope this would be a viable option, as I refuse to install Windows on my computer, and I am very intrigued by this program.

toucannyc: Will the program work for macs? You could also make a palm version. I think this is a great idea.

nachtrabe: It doesn't look like it would be that difficult to put together something like this for the Mac, particularly using Core Data. I don't have time right now, but might be convinced later if there is sufficient demand and no one else does it.

toucannyc: I found a program for mac called Jacoto. It's really great! It does all the things you're talking about. I don't know if there is a Windows version, but you can dowload the mac version for free. Is it the same thing that you'er trying to make?

Alejo Jacoto is available for Windows too. It's a Java based program. It has some nice options like statistics, filtering... though it has some bugs too

Requests

• I am using a german version of Microsoft Windows. I have access to a german Win98, Win NT4.0, Windows 2000 and XP. After the first version is finished, I am looking for someone who can test it on a (a) american version and (b) asian (japanese) version of Windows.
• I am really not good at graphics. If someone has an idea for a cool looking program icon or better tool bar buttons, please let me know.

What would you like your graphics to express? Maybe a go bowl would convey the idea of collecting? -- Sebastian

Contact

If you have an idea or suggestion, please add it to the "suggestions" section above. You can also leave me a message at my SL User page