Lizzie
Lizzie is an interface to LeelaZero (LZ for short), KataGo or – in principle – any other engine supporting the the necessary features of the GTP protocol as extended for Lizzie/LZ[3].
Table of contents |
Lizzie 0.7 (October 2019) using KataGo New: score graph (purple), ownership expectation for each point (greyscale)
Lizzie 0.6 (May 2019) using LeelaZero
How it works
See Uberdude's videos on youtube:
PJT: Hallo Uberdude, thanks for these videos. The second one breaks off with you starting to explain about changing networks: does that mean there is a third one due?
The screen
- Window title: Command line; number of playouts made in current position; visit/second.
- Note that the number of playouts need not begin at zero, as some may have occurred while analysing the previous position.
- Main board:
- Colored circles indicate moves LZ has analysed or is analysing. These contain numbers:
- At the top: the winning percentage for that move based on all the variations it has explored from there.
- In the middle (KataGo only): the estimated score.
- At the bottom: the number of variations it has explored from the move.
- The color of the circle shows how good it finds the move based on number of visits or (toggled with ‘k’) win rate:
- Cyan means: this is the move it has explored most, hence is its favorite move due to higher confidence with a deeper exploration. It also has a red ring around it when it is not the most heavily explored move and so another spot has a blue ring.
- Green means: these are moves with high win rates, so it's going to explore these some more.
- Red means: it has explored these moves as initial candidates but they give comparatively low win rates, so it has stopped exploring them.
- A blue ring around one or more spots highlights those with the highest win rate but not the deepest exploration. These are not necessarily those it believes best since a shallower exploration means lower confidence in the outcome.
- Hovering over each candidate shows the most likely variation it finds from that move, i.e. best play for both (in terms of winning percentages).
- With KataGo, you can choose to include a greyscale indication of how likely each point on the board is to be territory for either player. (This can be shown on the main board, the small board, both or neither.)
- Colored circles indicate moves LZ has analysed or is analysing. These contain numbers:
- Upper left:
- Black and white prisoners.
- Current win-rate estimate for Black and White (the broad horizontal black/white bar).
- The result of the last move.
- The win-rate graph, showing how the balance has shifted during the game.
- Red marks on the 50% line indicate blunders.
- Click on the graph to go to the corresponding move.
- Press ‘h’ to toggle between win-rate and handicap stones.
- With KataGo, this also shows how its estimate of the score has varied.
- Confusingly, there appear to be two estimates of the current score, often significantly different, one next to the score graph for the current move and one just above the graph in the middle. PJT: Could someone explain these, please?
- Lower left:
- The variation that LZ currently considers best.
- Bottom left: Whether “pondering”[1] is turned on or off; name of the file containing the neural network weights.
- Upper right: The game tree – click on a node to display the position on the board.
- Lower right: comment on the current position, as loaded from an SGF file.
For help
Hold down the ‘x’ key for a list of command keys (though not for the possibilities of clicking on the screen)
Hidden commands
Looking at the source[4] reveals a few commands that are not in the help text:
- ‘d’ toggles ‘dynamic komi’ if your LZ configuration supports it.
- ‘h’ toggles between showing win-rate and showing the advantage in handicap stones. This applies to the win-rate graph and the evaluations of the moves on the board, and can be particularly useful for handicap games.
- ‘k’ toggles between colouring moves by win-rate or by number of visits.
- ‘q’ ‘Open online dialogue’
- ‘y’ (in help, but unclear) Cycles through highlighting Black’s, White’s or nobody’s moves in the game tree.
- ‘,’ ‘Play best move’
- If pressed while hovering with the mouse over a rated move, Lizzie plays the entire sequence shown starting with that move.
- etc. To do: find out more from code.
Command line
The command line accepts a single optional undocumented argument specifying an SGF file to be loaded when Lizzie starts, e.g.:
java -jar lizzie.jar file.sgf
To analyse a game
- Press ‘o’ to open an SGF-file.
- If there was a handicap of several stones, press ‘h’ to show the advantage in handicap stones.
- You can let LZ analyse the whole game while you do something else, or move for move while you play through the game:
- To analyse move for move:
- Press the space bar to start “pondering”[1].
- Press the down arrow to move forward through the game and the up arrow to go back.
- Wait long enough at each position for LZ to make up its mind; this is very dependent on your hardware, but you should at least wait for the win rates to stabilise. (PJT can anyone suggest how many lines it should play out to reach say 1d or superhuman strength?)
- Once LZ has analysed a position, it saves the results, but it takes a while for Lizzie/LZ to consider and retrieve them if you return to the position.
- To analyse the entire game:
- Press ‘a’ to start the analysis.
- Say how many playouts you want for each move.
- You may want to experiment a little to find a reasonable value for this, which depends heavily on your hardware and the network installed.
- If you stay to watch, you should see Lizzie going through the moves; it goes on to the next move when the playouts in all moves considered reaches the limit you specified.
- If you want to stop it, just press ‘a’ again, otherwise wait till it reaches the last move.
- Once it reaches the end, it continues trying moves in the final position until you stop it (which is fairly futile but also fairly harmless).
- Once you stop it, you can click on big swings in the win-rate graph to jump to those positions and see the mistakes that caused the swings.
- Analysis always starts at the position currently shown, so you can analyse part of the game more deeply by specifying more playouts/move.
- Since LZ remembers its results, it only has to do the difference in playouts. E.g. if you analysed at 1,000 playouts and ask for 1,500 it only has to do 500 more for each move.
- If you click on the win-rate graph while it is analysing, it jumps to that position and continues analysing from there.
- You may also want to stop it before the end and finish the analysis with fewer playouts/move.
- To analyse move for move:
- Once you have analysed enough, you may want to save the SGF-file, as Lizzie includes (some) analysis results for each move.
- Though you might want to save the entire analysis to save time later, this does not seem possible: if your restart Lizzie, you need to reanalyse all positions you want to know about.
- Warning You may lose information from the original SGF, so consider saving it under a different name
- Press ‘s’ to save
- This is the (rather verbose) information included per move:
- The win-rate after the move and swing (98.7% (0.1%)) are replace by handicap stones if ‘h’ was used.
- The long name indicates the file containing the network used.
Move 2 98.7% (0.1%) (f178a6e01bcb3d611963da9a9ee29920904ed148ca095928fc9a0f936d162d4e.gz / 219 playouts)
Handicap games (warning)
Both analysing and playing handicap games can give confusing results, as LZ has learnt to play level games.
When analysing it is also likely to suggest lines leading to a complicated fight which the weaker player would prefer to avoid. It sometimes also chooses or suggests plays on the first line although it is unclear if they really are worthwhile or LZ has just lost the plot.
LZ, since it aims to maximise the win rate rather than the score, may also happily suggest sacrificing substantial groups in order to be even more sure of a narrow win, where the weaker player would be much better advised to hang on to all the points they can.
PJT: Are there perhaps networks trained so as to make them more helpful to weaker players (if that is possible)?
For analysing games between very unequal players on high handicaps, the win rate is often not very useful: Black will start with a very high probability of winning (above 99% for 6 stones), and gradually lose his advantage, but this will have relatively little effect on the win rate unless and until the game becomes roughly level. At this point the win rate will plunge dramatically to about 50% and, unless Black is able to hold his own, it will continue down to near 0%. To deal with this, use the ‘h’ command to show the advantage in handicap stones or (from version 0.7) use KataGo as the engine and attend to the expected score instead.
Some remarks (about LeelaZero)
- White always starts with a 55-45 advantage, presumably because of komi being too high (?)
- LZ maximizes the probability of winning, not the score
- It is generally accepted that small differences in the winning percentages do not indicate that a move is definitely better and only drops of 3-5% are significant enough to speak of a mistake
- As Uberdude convincingly shows, LZ sometimes doesn't consider a candidate, which, if shown by the human analyst, changes her mind about a certain position
- A known defect of AI like LZ, are ladders, which are a one way street for humans, but the random exploration of continuations tends to obscure the AI's judgment
Bugs
In version 0.6
- When a game is loaded, the function ‘i’ = “edit game information” does not display the information from the SGF-file.
Wishes
- When reviewing a game, Lizzie (or LZ?) should always do some analysis of the next move actually played, for comparison purposes.
- Should be possible to roll out and unroll the variations for each move one move at a time.
- Should be possible to hide all the coloured circles (to make reading easier).
Installing Lizzie
Download at https://github.com/featurecat/lizzie/releases and follow the instructions there. It includes a copy of LeelaZero and of KataGo, and everything should run pretty much out of the box under MS Windows.
Check the article on LeelaZero for tips on installing it and setting it up: especially on Linux, you may need to install a driver for your GPU (not without risks)[2] or to install or build a CPU-only version.[2]
You may also want to use a stronger network.
Notes
[1] “Pondering” refers to the function that lets LeelaZero analyse positions when it is not its turn to move (PJT Hope this explanation is correct!)
[2] See LeelaZero / Patrick Traill on Linux for the sad tale of how I got the GPU running but ended up with a predominantly dysfunctional laptop battery.
[3] Lizzie uses two GTP commands, lz-analyze and lz-genmove-analyze, which are extensions of the GTP protocol, to communicate with LZ and one extension, kata-analyze, for KataGo. (The convention is to name the prefix for the particular program the extension is for.)
[4] https://github.com/featurecat/lizzie/blob/master/src/main/java/featurecat/lizzie/gui/Input.java