The Robotic Solution to Bringing Online Go Into The Real World

CLACK

There is nothing quite like the sensation of placing a stone on a wood goban. Playing face to face, hands on stones on board, is undoubtedly the greatest joy of Go. The awesome reality of the our time is that a large number of online go servers exist which allow instantaneous gameplay and communication with players of all skill levels, all around the world. This opens up huge opportunities for new players to rapidly gain highly varied experience which simply cannot be found by traveling to their local go club once a week. Experienced players, even professionals, can log on to relax and play teaching games or challenge each other.

However

The sensation of placing a cursor on a computer screen and clicking, whether or not the particular server endows you with an audio clip of a stone clacking onto a board, can never be quite as satisfying as real gameplay.

I believe the solution to bringing online players into your home and onto your goban with real stones lies in robotics and software integration.

I call it Robotic Substitution Gameplay

Sounds difficult and expensive.

Well, that's mostly because it is somewhat difficult, in the sense that a very specific skillset is required, and could be expensive. However, thanks to the advent and rapid expansion of open source microcontroller platforms such as the Arduino, which can be purchased pre-built from a number of manufacturers or built from components costing less than a good steak at a fine restaurant, even moderately complex robots can be built on a modest budget. Bearing in mind that the functions required to pick up and place a stone, this is very good news. Even better news is that a number of startups and companies exist that manufacture kits based off of open source materials which will allow the simple assemble of pre-built components and the download of potentially free open source software to create a fully functioning, go playing, robotic arm.

I believe the ideal Go playing robot arm is a multiple-articulating arm with a swing base and a vacuum suction gripper. Anything similar to the UFactory Uarm could do the job, depending on the exact size of the player's goban. Relatively simple hardware. A hopper could be built using a vertical piece of pipe and a slanted piece of hard plastic to automatically gravity-feed pieces to a repeatable position programmed to be taught to the arm. The arm would pick up stones from this hopper to place on the board. The arm can also be taught the location of a prisoner pile, allowing it to remove captured stones from the board, adding to the sense of having a real opponent facing you.

The next, and greater, challenge is developing a way for the computer to sense the stones you have played on the goban without having to manually enter them into the computer. This additional action would invalidate the entire prospect of this project, which is to allow the player to play online go by only manipulating stones on a physical goban. I believe the solution is the use of a camera, whether an inexpensive webcam or any non-fisheye lens equipped camera. The camera would be mounted above the goban, facing directly down. Upon the player, placing a stone, they would then hit the space bar on their computer keyboard, not unlike hitting the timer to signify the end of their turn. Hitting the space bar would cue the computer to capture a photo through the downward facing camera. This image would be run through software designed to identify the positions and colors of stones on the goban based upon their contrast with the wood, and map their placement. Since only one stone is placed during a particular turn, the map would be compared with a map created from the players previous plus the robots turn and any resulting captures, and the coordinates of the singular new stone of the players color would be transmitted through the server as the players move. Any resulting captures would be noted by the server, and transmitted as a note to the player to ensure that the proper stones had been removed. The computer's map of the board would then be augmented with the absence of any captured stones in order to facilitate continued accurate tracking of the players moves. This cycle would repeat ad nauseum until the completion of the game. In this way, the player would have no physical interaction with the computer as a function of communicating their move except in a way identical to the existing necessity of striking a time clock as is necessary in tournament play, creating a real face to face experience regardless of geographical differences between the players.

The greatest challenge remains in the programming. I, admittedly, am not a programmer, however I believe I can outline the processes which I believe would be necessary to seamlessly integrate an existing online server to robotic play in real life.

-My preferred Go server is the Kiseido Go Server (KGS), so for the remainder of this article, I will be referring specifically to integration with gameplay hosted on KGS.

Notwithstanding the above discussion of imagery recognition software, these are what I believe to be the necessary steps to integrate robotic substitution play.

• Modification of the CGoban3 client app
  • Rebotic Substitution Gameplay option and tools
    • Drop down menu
    • Pathways for the communication from CGoban through USB connection to the microcontroller
    • Mouse or touchpad-driven manual arm manipulation to assist in teach function
    • Coordinate commands to be sent from CGoban to the microcontroller
    • Teach functions to define hopper, board coordinates, board height
    • Preference control to allow elimination or retention of goban in computer screen display
  • New interface which eliminates displayed Goban, but retains chat function
  • Notification function for identifying captured stones with images
• Microcontroller programming
  • Teachable functions to define hopper location, board coordinates, board height
  • Manual manipulation capability, directed by CGoban's tools, to assist in teach functions
  • Ability to receive goban coordinate information from CGoban, translate that to XYZ coordinates based on learned location of physical goban
  • Ability to learn goban height and plan movement paths to place stones without hitting the goban or other stones

A proposed walkthrough of the user experience using Robotic Substitution Gameplay on KGS

RSG setup

A primary RSG drop down menu from the top border of the dialog box would be created. In that list would be a primary option to enable or disable Robotic Substitution Gameplay as a whole. Enabling RSG would not change any view of the rooms or the CGoban3 app until a game was confirmed and begun. Enabling RSG in the drop box allows the selection of a "Tools" or "Options". Selecting "Tools" would open an independent dialog box. The top-left option in tools would be a "select a USB device" drop down menu under the text "Identify USB Arm." This drop down would show any USB devices attached to the computer and allow the selection of the RSG robot, which should already be in its operational location relative to the goban and hopper, as well as plugged into the computer. Upon the completion of this selection, a "Teach" option should become available on the left, under the. Selecting "Teach" should open a "Teach" wizard which enables at designated times, manual control of the arm using the mouse, touchpad, or keyboard. CGoban3 would have to programmed with direct action commands to the microcontroller in order to manually manipulate the robot in response to user inputs.

• This wizard should direct the user to first manually direct the arm directly over the hopper, down into contact with a stone, and actuate the vacuum gripper, gripping the first stone. Once this has been successfully completed, a "Confirm Position" button, available at the bottom of the wizard should be pressed. This action should save, within the microcontroller, that particular articulation and XYZ coordinate of the gripper as the location of the hopper.
• The wizard would then progress to the next screen, directing the user to move the arm, still gripping a stone, to a corner spot on the board, and down within a close distance of the board. The specific distance would be specific to the operational characteristics of the vacuum gripper in so much as it would reflect a distance far enough as to not interfere with its ability to drop the stone, and not so far as to cause the stone to fall an unnecessary distance. Once the user has manually positioned the arm in such a place, the user should select an "In Position" button on the dialog box. Selecting this option would drive the robot to automatically drop the stone. The user will assess if the stone drop height was appropriate and if its placement was accurate. If not, selecting "No, Try Again" would restore manual control, allow the user to pick up that same stone, and place it in a better spot to retest. If the initial test was successful, with no undue drop, and an accurate placement of the stone on the intersection, then the user would select "Confirm."
• Upon the selection of "Confirm," the wizard would prompt the user to manually move the arm back to the hopper, pick up another stone, and hold that stone one inch above the stone they had just placed on the board. Once the gripper is holding the second stone one inch above the first, the user would select "Confirm." This Z-axis height would be set as the minimum Z-axis value of the gripper during any travel in X or Y, ensuring that the arm, either while holding a stone or while empty-handed, does not strike or move any existing stones on the board. Once this value is confirmed, the wizard would prompt the user to hold that second stone directly above the corner point diagonally opposing the first. Upon selecting "In Position," the wizard would automatically manipulate the gripper to the correct Z-axis altitude over the board using the Z-axis data from the confirmed drop position of the first stone, and drop the stone into place. If this procedure is complete and correct, the user would select "Confirm." If this drop was incorrect because of a warped or other than flat board surface causing an incorrect Z-axis altitude, the user would likely have to restart the wizard and adjust the initial drop height to compromise across the goban. If the drop was incorrect because the stone was not correctly placed on the intersection, the user would select "Try again."
• Once the user has confirmed the second drop, the wizard would ask what size board was being used, the options available being 19x19, 13x13, or 9x9. Based on the XYZ coordinates of the opposing corners, and this knowledge of the total number of equally spaced intersections available, the microcontroller should calculate the XYZ coordinates of all remaining intersections.
• Once the board size is selected, the wizard will prompt the user to place the vacuum gripper in contact with that second corner stone and grip it. Once that has been completed the user will select "Confirm." This will set the Z-axis altitude necessary for the gripper to be in contact with stones on the goban that it needs to remove due to captures.
• Once this is confirmed, the wizard will prompt the user to manually manipulate the arm, still gripping the stone, over the container or location the user wishes it to use as its prisoner pile. Once the user selects "In Position," the gripper will drop the stone and request confirmation. This confirmation can be given by the user or the step can be redone with the same stone as in previous steps.
• Once confirmation has been given, the wizard will automatically manipulate the arm up to its minimum Z-axis altitude, over to the first stone placed on the board, and down into contact with the stone. The gripper will automatically grip that stone, lift it from the board to Z-axis travel altitude, and move over the the center, or tengen, space on the board. The arm will lower to drop altitude, place the stone on tengen, then raise, move to, and articulate down towards the hopper without gripping a new stone. This entire sequence will happen with no further input from the user as a confirmation of all registered XYZ coordinates and the microcontroller's calculation of tengen's position. The user will be prompted to confirm the success of this sequence. If this sequence fails, the entire wizard will restart upon selection of a "Failed" button on the wizard. If this sequence succeeds and the stone is placed properly on tengen, the user will select "Confirm."
• The wizard will then relinquish manual articulation of the arm to the microcontroller's automation. The wizard will send a command to the microcontroller that the stone on tengen has been captured, in the same format as it would be supplied from RSG-enabled CGoban3 had that message come from KGS' server. The arm, under the automatic direction of the microcontroller rather than manual manipulation by the wizard, should move to tengen, pick up the stone, place it in the prisoner pile, move to the hopper, and pick up its next stone, all while following all operational parameters established during the teaching wizard. The user will be given a final option to "Confirm" or designate that it "Failed." Failure would clear all data and restart the wizard, confirmation would close the teaching window.

Once the microcontroller has been taught it's operational parameters, the user must go back into the RSG drop menu and into "Tools." On the top right, under the text "Identify Detection Camera" would be another "select a USB device" drop menu. The user must select the camera that is already in place vertically above the board. Selecting the connected camera would make available a "calibrate" option. Selecting the "Calibrate" drop box would open a calibration wizard.

• The calibration would first direct the user to fine tune the placement of the camera so that the goban takes up as much of the camera's view angle as possible. This would be accomplished by a live camera view displayed in the wizard window. Once the user has placed the camera in optimal position, selecting "Confirm" completes the step.
• Upon confirmation, the wizard would display a warning that the camera cannot be moved from this point forward, especially after completion of the wizard. Any movement of the camera would invalidate any calibration previously completed and may, in the course of a game, cause faulty play coordinates to be transmitted out on KGS. Users must select "I understand" in order to continue.
• Once the user confirms understanding, the wizard would re-open a live view from the camera and direct the user to click directly upon two opposing corner intersections on the board. Once this is completed, the wizard will overlay a square on the live image and ask for confirmation that this square sits seamlessly on the exterior playable border of the board.
• Once the playable border is confirmed, the overlay would add two lines intersecting directly on the center space, or tengen, and again ask for confirmation.
• Once confirmation of the accuracy of tengen has been give, the wizard will close the camera view and ask what size the board is, options available being 19x19, 13x13, or 9x9.
• Once the board size has been selected, the wizard will show a live view of the board with an appropriate grid overlaid and designate 4 hoshi points to have white stones placed on them, and 4 hoshi points for black stones. Once the user has placed these stones in accordance with the instructions on the wizard, they will hit the space bar. Hitting the space bar causes the program to create parameters to recognize stones of both colors on the board in the current, specific lighting through shade and contrast.
• After completing this task, the live view will disappear and the user will be directed to place five stones of each color randomly throughout the board. Once this is completed, the user hits the space bar. Hitting the space bar will cause the computer to capture an image, analyze and determine the coordinates of all ten stones. The wizard will then produce a diagram of the board with all ten stones of the proper color in the proper place. If the diagram produced by the wizard is correct, the user hits "Confirm" which ends the wizard. Hitting "Calibration Failed" causes the wizard to clear all existing calibration and restart.

At this time, all necessary calibration would be complete, and the user is ready to configure their gameplay experience.

Configuring options

Under the primary RSG drop menu, below "Tools," an "Options" selection would be available. Selecting "Options" would open a new dialog box. This would have a few option sets available:

• Full-Time Screen Goban (Yes/No)
• Timer Size (Small/Large)
• Chat Window (Visible/Hidden)
• Capture notifications (ON/OFF AUDIBLE/MUTED)

Enabling the Full-Time Screen Goban option would result in the display on screen being identical to the one shown during regular gameplay with RSG disabled. If the Full-Time Screen Goban is enabled, all other options are unavailable. If the Full-Time Screen Goban is disabled, the display morphs into a chat window and a timer. The Small Timer option creates a screen which shows the observers to the game, a chat window, and the timer more or less identical to the typical display on CGoban 3, minus the displayed board. Large Timer option makes a large parallel pair of timers indicating the users and the opponent's available time. It would look like the display on a tabletop competition timer and would dominate the screen with a dark background and light text. The Chat Window could be visible in its appropriate size for the other option selected, or not, per the selection of the user. Capture Notifications would open a pop-up dialog box with a goban diagram highlighting the empty spots of stones that had been captured by the user's move. Notifications of capture by the opponent would be irrelevant, because the robot would have removed the opponents captures. The Audible notification option enables or disables a tone that sounds when a capture notification arises. If capture notifications are off, then the Audible/Muted option is unavailable.

At this point, RSG would be completely configured, and it's time to play!

Gameplay

On CGoban 3, viewing rooms, uploading game offers, receiving offers, and accepting opponents would be unchanged. The only additional task necessary for an RSG user would be preparing a hopper of the appropriate color stones, and having it in place before your opponent plays their first move, and viola, you are playing online go on your real, unmodified goban with your real, unmodified stones, against anyone, anywhere in the world! There is a slightly higher risk of some form of malfunction, so at any point, the option to cancel RSG usage, even in the middle of an active game, and finish through typical means would be available, but perhaps the user should inform his opponent as well that they are using an RSG system. --- Summary

I believe this is an inevitable next step in online server hosting of not just Go, but virtually all boardgames. At this time, even if the software were created and uploaded as free and open source, RSG could still be cost-prohibitive. It also could simply require more time than one might be willing to dedicate to creating their own system. I currently do not have the expertise, time, or resources to make RSG a reality, even for just myself, which is why I have shared this vision here, to hopefully inspire a collective effort to make what I believe is a really cool concept, and an opportunity for Go to become known amongst the next generation of tech-savvy millenials.

Regardless,

I believe Robotic Substitution Gameplay is the inevitable next step in bringing the unlimited potential of online Go to bear with the authentic experience of stones on wood. When it happens is up to us.

Any programmers, arduino builders, or robotics experts are encouraged to comment. I am none of those things, so if I proposed something rather preposterous, or outright impossible, then let me know!