This is an experiment in tokenmaxxing where I checked what happens if I just let it run with minimal interventions
I asked a free AI coding agent — Meta Spark 1.3, running through OpenCode — to build a backgammon bot that could eventually beat the best existing engines. Its small network became much better at choosing moves. More varied games helped, a different board representation helped, and deeper supervision helped. But the part I most wanted to work kept stalling: let the bot play, learn from its own decisions, and produce a stronger successor.
It went through search algorithms, teacher models, training targets, larger networks and the doubling cube. But why so much useful work produced so little sustained self-improvement?
The plan was to make the bot its own teacher
AlphaZero combines a network with search. The network suggests promising moves and estimates the value of positions. Search uses those estimates to examine continuations. The resulting games provide training targets for the next network. If search consistently finds better decisions than the network would make alone, perhaps training can turn that extra thought into a better starting point for the next search.
Backgammon adds chance. After choosing a move, you don’t know the opponent’s next roll. Two dice have 21 distinct unordered outcomes, so search needs to account for their probabilities as well as both players’ choices.
Stochastic MuZero showed that this approach could produce a strong backgammon player. It also learns the environment’s dynamics. The game rules and dice probabilities were already known, so the agent used the actual simulator instead.
It trained a network to imitate Sage, an existing engine, then used it to guide search. That supervised start was supposed to get to the interesting RL experiments sooner.
More search didn’t reliably mean better moves
Its first question was whether search could improve the network at all. It measured regret on a set of positions: the expected score a reference evaluator thought was lost by choosing its move instead of the reference’s preferred move. Lower was better. This gave a faster signal than playing thousands of noisy games, though it measured agreement with a fallible reference rather than tournament strength.
Small searches sometimes helped. Adding more simulations didn’t produce a clean improvement curve. A network that gave poor estimates to the search could still persuade it to investigate the wrong moves.
It compared PUCT, the search rule used in the AlphaZero family, with Gumbel-based search. It tried sampling dice, covering the outcomes more evenly, and expanding all 21 possibilities. It also compared choosing a whole turn at once with choosing one checker movement at a time.
Covering dice outcomes more evenly helped in one comparison. Expanding all of them cost roughly six times as many value evaluations without a corresponding gain. Complete-turn search performed poorly at small budgets, and Gumbel didn’t give a winning configuration.
These were useful constraints on the implementation. They didn’t establish that those algorithms were unsuitable for backgammon. It was comparing ways of spending a small budget using an imperfect evaluator.
A better move preference could make search worse
The network had two jobs: suggest what to play, and predict how good the position would be. The agent tried improving their training targets separately.
Giving the value prediction deeper teacher evaluations barely moved raw play. Giving the move preferences deeper evaluations reduced regret from about 0.0810 to 0.0752.
Then search made that stronger policy worse at every tested budget.
I had expected better ingredients to make a better system. But a stronger initial preference also gives search more to lose when its estimates lead it away from a good move. Improving one component changes what the rest of the system must do to help.
It tried retuning exploration and sharpening the preferences. Neither rescued that model’s search. At that point, another clever search modification was less convincing than checking whether the network had seen enough different games.
The first clear breakthrough came from more varied games
It collected more trajectories, including plausible and suboptimal play, then trained on nested subsets while keeping the architecture and recipe fixed.
| Training games | Raw regret with deeper policy targets |
|---|---|
| 250 | 0.0625 |
| 500 | 0.0461 |
| 1,000 | 0.0337 |
At 1,000 games, the same search reduced regret further to 0.0263 with eight simulations per checker decision. This was the first model to pass the statistical gate for trying self-play.
The result also changed how I thought about dataset size. A backgammon turn can become several training rows, and multiple move orders can reach the same board. Those rows help teach the network how to complete a turn, but they don’t represent independent strategic experience.
Its later retained dataset contains 733,269 rows from 1,048 source games. The row count sounds much larger than the variety behind it.
The benchmark figures here are historical development measurements. The agent repeatedly returned to the same small set of positions, so they shouldn’t be read as independent confirmations of every successive improvement.
Self-play made the first successor worse
The successful model generated 1,536 self-play games and roughly 248,000 decision samples. The agent trained one successor entirely on that experience and another with the supervised examples mixed back in. Both went backwards: raw regret rose from 0.0337 to around 0.040, and search stopped improving either one.
The recovery attempts landed in the same place. Anchoring the value predictions to the old network reduced drift without producing a better player. Rotating opponents across checkpoints beat replaying copies of one model and still left the baseline unbeaten. Mixing search preferences with the old network’s preferences recovered raw regret to 0.0349, close to the starting model, but turning the search contribution back up made raw play worse again. Each attempt bought a network that search helped more, or a better raw policy, never a stronger system than the one it started from.
A further generation from the best recovery model pushed raw regret to 0.0381, with training loss falling while validation performance worsened.
At a few simulations, a search result is a sparse and noisy teaching signal. The student approximates it, visits different positions as a result, and hands those positions to the next student. Errors travel that loop as easily as improvements. Game outcomes add their own noise, since a good decision can still lose to an unlucky roll, and replaying a small set of games for many epochs doesn’t produce more independent outcomes.
The agent changed the teacher, then how the student saw the board
With the loop stalled, it went back to supervised learning and added WildBG as a second teacher, useful because its strengths weren’t the network’s. Splitting positions into contact, where players can still hit each other’s checkers, and races, where the only remaining problem is bearing off, put most of the regret in contact. Imitating WildBG improved race play and barely touched the contact gap. Oversampling difficult backgame positions made overall play worse: focusing the dataset on one weakness changed what the network practised everywhere else.
Borrowing from established backgammon evaluators, the agent stopped encoding a point as a checker count and started exposing whether it held one checker, two, or a larger stack. Contact and race also got separate networks. That combination, plus deeper Sage supervision, brought raw regret to 0.0250 in a checkpoint of about 221,000 parameters.
At larger search budgets it played WildBG roughly even. Against Sage at two-ply it won 31 of 100 cubeless games, on a harness the agent later found needed repairs. Re-run on the repaired harness: 36 of 100 cubeless, and 48 of 100 cubeful with Sage’s own cube decisions. Better, and the gap stands.
The second RL attempt stalled more quietly
With the stronger network, the agent ran the loop again. One generation used 832 games at 16 simulations, the next 768 games at 32. Both finished around parity with WildBG, and nothing compounded. Larger networks, deeper teacher labels and extra training on positions where the model disagreed with Sage produced no reliable replacement either.
The tempting conclusion was that the ideas had run out. The narrower one fits the evidence better: these recipes, at these data sizes and search budgets, hadn’t produced another improvement. A larger model failing on one dataset says nothing about what it would do with more varied experience, and two flat generations say nothing about what a hundred would do.
The cube became a useful detour
Alongside checker play, the agent worked on the doubling cube. Before rolling, a player may offer to double the stakes, and the opponent accepts or concedes. How large wins and losses get now depends on a second set of decisions.
That work ran from simple yes/no heads to networks predicting the equity of each alternative, and it taught me how representation, target choice and data collection interact. Recording passed doubles mattered, because accepted doubles alone are an incomplete training set. It was also a second supervised project, which made the work look busier without answering the self-play question.
A later audit, the agent reading back over its own trail, turned up four defects: cube decisions taken partway through a turn, a wrong Sage probability conversion, benchmark and live move legality that disagreed, and an exploration setting that computed noisy preferences and never delivered them to search. All four are fixed now and the affected matches re-run. The numbers changed without changing the overall picture.
Against WildBG the edge survived repair and shrank: three seeds, 1,200 games, +0.56 points per game with the confidence interval just clearing zero, down from the +1.25 the buggy loop had claimed. Partway-through-turn doubles had been inflating the score. Against Sage, the cube recovers win rate, 36% cubeless to 48% cubeful, but not points, because a stronger doubler selects against exactly the takes the bot wants to accept.
Two attempts to retrain the take decisions on corrected games both looked better on the frozen benchmark and failed live. Every candidate now lands in its 92-94% band while live results diverge. The benchmark had stopped telling good take heads from bad ones: the faster-signal problem from the move-choice section, again.
Why several weeks of work felt slower than expected
Only some of the work tested the original hypothesis about self-improvement. The rest compressed a teacher, guided search, collected varied experience and predicted cube decisions.
It also asked a small benchmark too many questions. Those positions came from 21 games, and using them to pick the next experiment made them part of development, whatever the word “test” in the filename claims.
Scale was the other gap. DeepMind’s published backgammon configuration specifies 1,600 simulations, 1,000 actors and eight million training updates. The action representation differs, so simulation counts aren’t directly comparable, but a few small generations are not that training regime.
Experiments were also slow enough to discourage the comparisons worth running. Game generation took long enough that another data scale became an expensive commitment. When the agent finally profiled it, the cost turned out to be an overloaded machine rather than enumeration or opponent queries.
The next milestone is a repeatable learning curve
Next is a comparison between the corrected loop and a plain temporal-difference learner, which gives a cheaper RL baseline and a direct way to study bootstrapping from future value estimates. A recent PureTD preprint makes that comparison interesting too.
Variety of games bought the first gain. Changing what the network was taught and how it saw the board bought the second. I want the third to come from the bot generating experience that makes its successor reliably stronger.