Chapter 26 · Section 26.2 · 100 min

Self-play

Self-play means training a program by letting it play against copies of itself and learning from the results. It produced Go and chess programs far stronger than any human, and it is the cleanest example of a system that improves on data it makes itself. This section shows how self-play works, why it works so well in games, where it breaks, and what is left of it when the game is replaced by language.

This chapter makes one argument, first stated in §26.1. A model can learn from its own outputs only when something tells it which outputs were good, and that something must be more reliable than the model's own guesses. In a game, the rules do this. At the end of every game you know exactly who won.

Here is how far that goes. In 2017 DeepMind trained a Go program, AlphaGo Zero, that started from random moves and was told nothing but the rules (Silver et al. 2017). It never saw a human game. After three days of playing against itself, it played AlphaGo Lee, the earlier program that had beaten Lee Sedol, one of the strongest players in the world. It won 100 games to 0. Its successor, AlphaZero, learned chess and shogi the same way.

Three things made this possible, and we build each one from scratch. First, the rules give an exact reward. Second, a copy of yourself is an opponent of exactly your strength, so you win about half your games and every game teaches something. Third, a search that looks a few moves ahead turns the network's current guess into a better move, and the network then learns to imitate that move.

We build all three on a game small enough to solve by hand. Then we see where the recipe breaks, in games where "better" goes in circles and in superhuman programs that lose to a trick a human player can learn. Last, we follow self-play into language models, which have the opponent but not the rulebook. That answers the question behind the section. Why did self-play produce superhuman Go players but not superhuman essay writers?

A game small enough to solve by hand

Take a pile of stones. Two players take turns, and on each turn the player to move removes 1, 2, or 3 stones. Whoever takes the last stone wins. This is one-pile Nim with a move limit. It is small enough to solve exactly, so every algorithm in the first half of the section can be checked against the true answer.

A position is the number of stones left, $n$ . We score each position for the player about to move, and call the score its value. It is $+1$ if the player to move wins with best play and $-1$ if they lose. With 0 stones the player to move has already lost, because the opponent took the last stone, so the value of 0 stones is $-1$ .

The key step is to look at each move from the opponent's side. Whatever position I leave, my opponent moves next. The game is zero-sum, which means one player's win is the other's loss. So a position worth $+1$ to my opponent is worth $-1$ to me, and I want to leave them the position with the lowest value.

The same recursion works for every game of this kind.

📐 Two-player zero-sum game and its negamax value

A two-player zero-sum game of perfect information has states $s$ , a set of legal moves $\mathcal{A}(s)$ in each non-terminal state, and a transition $s' = \operatorname{next}(s, a)$ . The players alternate moves. Both see the whole state, which is what perfect information means (poker, with its hidden cards, does not have it). Each terminal state has an outcome, scored for the player to move there as $+1$ for a win, $-1$ for a loss, and $0$ for a draw. The game value $V^\star(s)$ of a non-terminal state is defined by the negamax recursion

V^\star(s) = \max_{a \in \mathcal{A}(s)} \big[-V^\star(\operatorname{next}(s, a))\big].

A move $a$ is winning in $s$ if $-V^\star(\operatorname{next}(s, a)) = +1$ .

In plain words, my value is the best I can get, and what I get from a move is minus what my opponent gets from the position it leaves. The name is short for "maximum of the negation". That minus sign is the whole difference between a puzzle, where one player maximizes, and a game, where two players pull in opposite directions. The cell fills in the table for Nim.

Python · cell 1

Chess and Go fit the same definition, so in principle the same table solves them. Working backward from the end of the game like this is called backward induction. It proves that every finite two-player game of perfect information has a value and an optimal strategy for each player, an argument that goes back to Ernst Zermelo in 1913. In practice the table is far too big. Chess has about $5\times10^{44}$ legal positions and Go about $2\times10^{170}$ . Everything that follows is a way to approximate $V^\star$ and the best moves without the table.

Tree search looks ahead only where it matters

Monte Carlo tree search (MCTS) looks ahead from the current position without building the whole game tree. It grows a small tree of positions below the current one and estimates how good each move is by averaging the results of many sample walks through that tree, which is what "Monte Carlo" refers to. §18.6 used it to search over the steps of a written solution, where only one player chooses. Here we build the whole algorithm for a two-player game, the version inside AlphaZero.

One walk down the tree is called a simulation. It starts at the root, the current position. At each position it takes the move with the best score. The score adds two things, how well the move did in earlier simulations and a bonus for moves that look plausible but have rarely been tried.

When the walk reaches a position not yet in the tree, it adds that position, asks for an estimate of its value, and carries the estimate back up the path. After many simulations, the moves at the root that were tried most often are the ones that held up best under the most scrutiny. Those visit counts are what the search returns.

Two functions guide the search. A policy gives every legal move a probability, and the search uses one as its prior $P(s, a)$ , a first guess at which moves are good. A value function $v(s) \in [-1, 1]$ is a guess at $V^\star(s)$ . In AlphaZero both come from one neural network. For now they can be anything, even a uniform prior and a value of zero everywhere.

📐 MCTS with PUCT for a two-player zero-sum game

Each node of the tree is a non-terminal state $s$ . For every legal move $a$ it stores the prior $P(s,a)$ , a visit count $N(s,a)$ , and a value sum $W(s,a)$ . The mean value is $Q(s,a) = W(s,a)/N(s,a)$ , with $Q = 0$ while $N(s,a) = 0$ . All values are for the player to move at $s$ . Write $N(s) = \sum_b N(s,b)$ for the node's total visits, and let $c_{\text{puct}} > 0$ set the weight of the exploration bonus. The root is expanded first. Each simulation then runs three steps.

Select. Starting at the root, repeatedly take the legal move
$a^\star = \arg\max_{a \in \mathcal{A}(s)} \Big[\,Q(s,a) + c_{\text{puct}}\,P(s,a)\,\frac{\sqrt{N(s)}}{1 + N(s,a)}\,\Big],$
breaking ties toward the first legal move, until the move leads to a terminal state or to a state not yet in the tree.
Evaluate. A terminal state gets its exact value. A new state $s_L$ is added to the tree with its priors and zero counts, and gets the estimate $v(s_L)$ .
Back up. Walk back along the path. At each edge $(s, a)$ , first negate the value, then record it, $N(s,a) \mathrel{+}= 1$ and $W(s,a) \mathrel{+}= v$ . After the negation, $v$ is the value for the player to move at $s$ .

After $n_{\text{sims}}$ simulations the output is the root's visit counts $N(s_0,\cdot)$ , which sum to $n_{\text{sims}}$ , and its values $Q(s_0, \cdot)$ .

The selection rule is called PUCT, short for "predictor plus upper confidence bounds applied to trees", where the predictor is the prior. §18.6 wrote its constant as $c$ . We rename it $c_{\text{puct}}$ , because in this chapter $c$ counts correct samples.

In plain words, go where the average outcome has been good, and also where the prior thinks a move is plausible and you have not looked much yet. Look at the bonus term. At a fresh node $N(s) = 0$ , so every bonus is zero and the tie rule picks the first move. After that the bonus grows like $\sqrt{N(s)}$ for a move you keep skipping and shrinks like $1/N(s,a)$ for a move you keep choosing. So every move with a positive prior is tried eventually, and as visits pile up, $Q$ decides.

Why the backup flips the sign

The negation in step 3 is the one line that separates search in a game from search in a puzzle. It is also the most common bug in hand-written MCTS.

Proof. The players alternate, so the player to move at $s_j$ is the player to move at $s_L$ exactly when $L - j$ is even. In a zero-sum game an outcome worth $v_L$ to one player is worth $-v_L$ to the other. ∎

Forget the negation and every node stores values for the player at the leaf. Both players then maximize the same number, and the search behaves as if the two sides were cooperating. A cell below shows the damage. From 2 stones, "take 2" wins on the spot, yet a search without the flip scores it $-1$ , because it scores the move for the player who just lost.

Four simulations by hand

Let us run the algorithm by hand from 5 stones, where the only winning move is to take 1. The prior is uniform, $P = \tfrac13$ for each move, and $c_{\text{puct}} = 2$ . The value function is weak but points the right way. It says $v(n) = +0.25$ if $n$ is not a multiple of 4 and $-0.25$ if it is. No simulation reaches the end of the game in this short run.

✏️ Four PUCT simulations from 5 stones

The root scores are $Q + 2\cdot\tfrac13\cdot\sqrt{N(s_0)}/(1 + N(s_0,a))$ for the moves take 1, take 2, and take 3.

sim	$N(s_0)$	root scores	path	new leaf, $v$ for its mover	root $N$ after	root $Q$ after
1	0	0, 0, 0 (a tie)	take 1	4 stones, $-0.25$	1, 0, 0	0.25, 0, 0
2	1	0.583, 0.667, 0.667	take 2	3 stones, $+0.25$	1, 1, 0	0.25, $-0.25$ , 0
3	2	0.721, 0.221, 0.943	take 3	2 stones, $+0.25$	1, 1, 1	0.25, $-0.25$ , $-0.25$
4	3	0.827, 0.327, 0.327	take 1, take 1	3 stones, $+0.25$	2, 1, 1	0.25, $-0.25$ , $-0.25$

In simulation 1 the new leaf has 4 stones. It is bad for the opponent, who must move there, so $v = -0.25$ , and after the flip the root's edge records $+0.25$ . In simulation 4 the walk enters the 4-stone node. All its edges are unvisited, so the tie rule takes 1, and the walk reaches 3 stones with the root player to move again. The leaf value $+0.25$ is negated once for the opponent's edge ( $-0.25$ ) and again for the root's edge ( $+0.25$ ).

After four simulations the visits are $(2, 1, 1)$ . As probabilities that is $(0.5, 0.25, 0.25)$ , so the winning move rose from the prior's $\tfrac13$ to $\tfrac12$ .

The search did not improve because it found a forced win. It improved because it looked one move ahead and asked the value function, which already knows more than a uniform guess. With more simulations, terminal states enter the tree and their exact values flow back up.

Visit counts become a policy

The root's visit counts are the search's verdict. AlphaZero turns them into a probability distribution with a temperature $\tau \ge 0$ ,

\boldsymbol\pi^{\text{search}}(a\mid s) \propto N(s,a)^{1/\tau}.

At $\tau = 1$ the probabilities are proportional to the visits. As $\tau \to 0$ all the mass goes to the most-visited move, which is how a program picks its move in a real match. In self-play, AlphaGo Zero used $\tau = 1$ for the first 30 moves of each game, so that games start differently, and $\tau \to 0$ afterward. We write $\boldsymbol\pi^{\text{search}}$ so that it never collides with the network's own policy $\pi_\theta$ .

Warm-up

Policy targets from visit counts

Write search_policy(visits, tau, legal), which turns root visit counts into the training target $\boldsymbol\pi^{\text{search}} \propto N^{1/\tau}$ .

visits is a float array of visit counts, and legal is a boolean mask of the same shape. Return a probability vector over the legal moves with exactly zero on the illegal ones.
tau is a temperature of at least 0. At tau == 0, return the argmax, splitting the mass evenly among tied most-visited legal moves.
If no legal move has any visits, return the uniform distribution over the legal moves.

Work in log space. Small temperatures overflow otherwise, since with tau = 0.005 the power $800^{200}$ is too large for a float.

import numpy as np

def search_policy(visits, tau, legal):
    visits = np.asarray(visits, dtype=float)
    return visits / visits.sum()

Tests, hints, verified solutions, and AI help, with Pro at $7.99 a month.

Next is the interview problem. Labs ask for MCTS on a whiteboard often enough that you should be able to write it without notes, sign flip included.

Interview

MCTS with PUCT for a two-player game

Write mcts(game, root, prior_fn, value_fn, n_sims, c_puct=1.5, eps=0.0, alpha=0.3, seed=0) exactly as the definition box above describes. It returns the root's visit counts N and mean values Q, both arrays of length n_actions. The inputs are these.

game has n_actions, legal(s) (a boolean mask), next_state(s, a), and terminal_value(s). The last returns the exact value of a terminal state for the player to move there, or None if the game goes on.
prior_fn(s) returns nonnegative weights over all actions. Mask them to the legal moves and renormalize.
value_fn(s) returns $v(s)$ for a new non-terminal leaf.

The tests check visit counts down to the last simulation, so follow these rules exactly.

Build a tree with one node per path, not one per state.
Expand the root before the first simulation, and give unvisited edges $Q = 0$ .
Break ties with np.argmax, which picks the first legal move.
Never expand a terminal state, and negate the value once per edge on the way up.
If eps > 0, mix random noise into the root's priors only, $(1-\varepsilon)P + \varepsilon\,\boldsymbol\eta$ , after masking. Here $\boldsymbol\eta \sim \operatorname{Dir}(\alpha)$ is a random probability vector over the legal moves. Draw it once, as np.random.default_rng(seed).dirichlet([alpha] * k) for the $k$ legal root moves, and place its entries on those moves in order (zero elsewhere). The AlphaZero part of this section explains why the noise is there.

The setup provides Nim (taking the last stone wins), MisereNim (taking the last stone loses), uniform_prior, and zero_value.

import numpy as np

def mcts(game, root, prior_fn, value_fn, n_sims, c_puct=1.5, eps=0.0, alpha=0.3, seed=0):
    N = np.zeros(game.n_actions)
    Q = np.zeros(game.n_actions)
    # TODO: select with PUCT, expand, evaluate, back up with the sign flip
    return N, Q

Tests, hints, verified solutions, and AI help, with Pro at $7.99 a month.

Here is our own version, which the rest of the section uses. It prints the root scores of the hand example, and they match the table.

Python · cell 2

What search alone can and cannot see

Python · cell 3

Search alone is excellent near the end of the game and useless far from it. With 300 simulations it puts 94 percent or more of its visits on the winning move up to 7 stones, and 60 percent at 11. From 13 stones on it falls back to about the prior's one third. The winning line there is deeper than the tree grows, and every leaf the search sees says 0.

This blind spot is called the horizon problem, and it is why AlphaZero needs a value network. A value function that is right about positions beyond the tree's reach extends what the search can see. The search, in turn, produces better training targets for the value function. The widget lets you watch both effects.

An interviewer who asks you to write MCTS is checking three things. Do you flip the sign at every level? Does the bonus use the parent's total visits? Can you say why the search needs a value function at all?

Why imitating the search makes the network stronger

AlphaZero trains its network to predict the search's visit counts. Why should that make the network stronger? There is an exact argument for one step and an empirical one for the whole loop.

Start with the exact part. Fix the opponent's strategy. From one player's side the game is then an ordinary Markov decision process, a sequence of states, actions, and rewards in which the next state depends only on the current state and action. §15.2 proved the policy improvement theorem for such processes. Let $Q^{\pi}(s,a)$ be the exact action values of a policy $\pi$ , the expected outcome of playing $a$ in $s$ and following $\pi$ afterward. The greedy policy $\pi'(s) = \arg\max_a Q^{\pi}(s,a)$ does at least as well as $\pi$ in every state. It does strictly better somewhere unless $\pi$ is already optimal.

In plain words, acting greedily on accurate values improves a policy. MCTS estimates those values at the current position, with more look-ahead than the network has. As simulations grow, its visits pile up on the moves with high $Q$ , because a move's bonus shrinks as the move is visited while its $Q$ does not.

How good is the estimate? PUCT descends from UCT, the upper-confidence rule for trees. UCT uses no prior, and its bonus is $c\sqrt{\ln N(s)/N(s,a)}$ . For UCT, Kocsis and Szepesvári (2006) proved that the value estimates converge to the true game values as the number of simulations grows. For PUCT with a learned prior and value there is no comparably clean guarantee. That search plus imitation keeps improving the network over millions of games is an empirical finding.

The size of the effect is measured in Elo, the rating scale of chess and Go (§22.7 derives it). A player rated $\Delta$ points higher is expected to score a fraction $1/(1 + 10^{-\Delta/400})$ of the points. So a 400-point edge means scoring about 10 points for every 1 the opponent scores.

In practice this means two things. A self-play program always plays its real matches with search switched on. And the search's output is a better label than anything the network could produce alone, which is the whole reason to train on it. An interviewer who asks "why does imitating MCTS help?" wants the policy improvement theorem and this gap.

🔬 The visit distribution solves a KL-regularized problem

Grill et al. (2020) made the policy-improvement reading precise. Let $\mathbf{q}$ be the root's search values and $\pi_\theta$ the prior. They showed that AlphaZero's visit distribution tracks the solution of

\bar{\boldsymbol\pi} = \arg\max_{\mathbf{y} \in \Delta_{|\mathcal{A}|}} \Big[\mathbf{q}^\top\mathbf{y} - \lambda_N\,D_{\mathrm{KL}}(\pi_\theta \,\Vert\, \mathbf{y})\Big], \qquad \lambda_N = c_{\text{puct}}\frac{\sqrt{N(s)}}{|\mathcal{A}| + N(s)},

where $\Delta_{|\mathcal{A}|}$ is the set of probability vectors over moves. The objective rewards high search values and charges for moving away from the prior. It is a cousin of the KL-regularized optimum of §17.1, with the KL taken in the other direction. Its solution is $\bar\pi(a) = \lambda_N\pi_\theta(a)/(\alpha - q_a)$ , where $\alpha$ (unrelated to the Dirichlet $\alpha_{\text{dir}}$ below) is the number larger than every $q_a$ that makes the probabilities sum to one. As $N(s)$ grows, $\lambda_N \to 0$ and the target moves from the prior toward the best move by search value. Search moves the policy as far as the evidence allows, and no further.

AlphaZero closes the loop

We now have all the pieces. One network reads a position and outputs two things, a prior over moves and a value. MCTS uses the prior to decide where to look and the value to judge new leaves. The program plays games against itself and chooses every move by search. Each position it meets is stored with two labels, the search's move probabilities and the final result of the game. Then the network is trained to predict both labels. A better network makes a better search, which makes better labels.

📐 AlphaZero

A network $f_\theta(s) = (\mathbf{p}, v)$ maps a position to move probabilities $\mathbf{p} \in \Delta_{|\mathcal{A}|}$ , with illegal moves masked, and a value $v \in [-1, 1]$ that predicts the outcome for the player to move. In self-play, each move is chosen by MCTS with PUCT, using $P(s,\cdot) = \mathbf{p}$ as the prior and $v$ for leaf values. At the root only, random noise is mixed into the prior,

P(s_0, a) = (1 - \varepsilon_{\text{dir}})\,p_a + \varepsilon_{\text{dir}}\,\eta_a, \qquad \boldsymbol\eta \sim \operatorname{Dir}(\alpha_{\text{dir}}),

and the move is sampled from $\boldsymbol\pi^{\text{search}}(a\mid s) \propto N(s,a)^{1/\tau}$ . Each position $s_t$ of a finished game is stored with its search policy $\boldsymbol\pi_t^{\text{search}}$ and the outcome $z_t \in \{-1, 0, 1\}$ for the player to move at $s_t$ . The parameters minimize, averaged over stored positions,

\ell(\theta) = (z - v)^2 - \boldsymbol\pi^{\text{search}\top}\log\mathbf{p} + c_{\text{reg}}\lVert\theta\rVert^2 ,

where $c_{\text{reg}}$ is the weight-decay coefficient ( $10^{-4}$ in AlphaGo Zero). The newest parameters play the next games (Silver et al. 2017, Equation 1).

In plain words, the value head learns to predict who actually won from here. The policy head learns to predict what the search chose here. A little weight decay keeps the weights small. Both labels come from the system itself, and both are better than what the network knew when it played the game. The outcome $z$ is exact, because the rules say who won. The search policy beats $\mathbf{p}$ , by the improvement argument above.

Four design choices and their reasons

Interviewers ask about each of these.

Visit counts, not just the best move. A distribution carries more information than one move, the way soft labels carry more than hard ones. It also keeps the search's uncertainty, since a move visited 400 times out of 800 is less certain than one visited 790 times. The $Q$ of a rarely visited move is a noisy average of a few leaves, while the visits already reflect how much the search trusted each estimate. And a one-hot target would collapse the prior. A move with a prior near zero gets almost no bonus in the next search, so it would never be explored again.

Noise at the root. Self-play learns only about moves it plays. If the prior gives a strong move almost no probability, the search rarely visits it, the visits never favor it, and the network never learns it. The fix mixes random noise into the root's prior with weight $\varepsilon_{\text{dir}} = 0.25$ . In each search a few random moves, possibly ones the network dislikes, then get a real share of the prior. Meanwhile 75 percent of the prior still comes from the network, so the search can overrule bad noise. The noise is drawn from a Dirichlet distribution $\operatorname{Dir}(\alpha)$ , a distribution over probability vectors. With $\alpha$ below 1 its samples are spiky, with most of their mass on a few random moves. AlphaZero scales $\alpha_{\text{dir}}$ inversely with the typical number of legal moves, using 0.3 for chess, 0.15 for shogi, and 0.03 for Go. That keeps $\alpha_{\text{dir}}$ times the number of moves near 10, so in every game the noise gives a real boost to a small random set of moves.

A temperature. At $\tau = 1$ the move is sampled in proportion to the visits, which varies the openings the network trains on. Later in the game AlphaGo Zero played the most-visited move ( $\tau \to 0$ ).

No gate. AlphaGo Zero kept a "best player" and replaced it only when a new network won more than 55 percent of 400 evaluation games. AlphaZero dropped this gate and always played with the newest network. That it still worked suggests that in these games the improvements are steady enough that an occasional worse network does little harm.

The loop is one case of a general template called expert iteration, which Anthony, Tian, and Barber (2017) developed independently on the board game Hex. An expert, here search guided by the network, makes better decisions than the apprentice, the network alone. The apprentice is trained to imitate the expert, and the improved apprentice makes the expert stronger. §26.3 shows that STaR and rejection-sampling fine-tuning for language models use the same template, with sampling plus an answer checker as the expert.

The two kinds of compute also trade against each other. Jones (2021) trained AlphaZero-style agents on Hex at many board sizes. On a 9 by 9 board, each additional tenfold of training compute removed the need for about fifteenfold of search compute at play time, with no loss in strength (§9.6 puts this beside test-time compute for language models).

📜 From AlphaGo to AlphaZero

AlphaGo (Silver et al. 2016) started from a policy network trained on 30 million positions from human expert games. It improved that network by self-play with policy gradients and trained a separate value network. Its search judged leaves by mixing that value with the results of fast playouts to the end of the game, in which a small, quick policy network chose the moves. It beat the European champion Fan Hui 5 to 0, and in March 2016 it beat Lee Sedol 4 to 1. That version is called AlphaGo Lee.

AlphaGo Zero (Silver et al. 2017, Nature) removed every human ingredient. It used one network with a policy head and a value head, no playouts, and only the rules. Its first run played 4.9 million games against itself over about three days, with 1,600 simulations per move (about 0.4 seconds). It trained on 700,000 mini-batches of 2,048 positions drawn from the most recent 500,000 games. After 72 hours it beat AlphaGo Lee 100 games to 0. A larger 40-block network, trained for about 40 days on 29 million games, reached 5,185 Elo. It beat AlphaGo Master, the version that had won 60 straight online games against top professionals, by 89 games to 11.

AlphaZero (Silver et al. 2017, preprint) ran the same algorithm on chess, shogi, and Go with one set of hyperparameters, apart from the root noise. It stopped using the board's 8 symmetries, which only Go has. It predicted the expected outcome rather than the probability of winning, so draws count. And it dropped the gate.

What the loop costs at scale

AlphaZero's hours of training hid a large fleet of chips. Its preprint reports the hardware and the game counts, and those numbers say where a self-play system spends its compute.

The counts in the box are enough to compare the two halves of the loop, playing games and training on them, by the number of network calls each one makes.

✏️ Where AlphaZero's compute went

Count network calls in the chess run. Assume a game lasts about 80 moves, counting each player's move separately. Then 44 million games hold about $44\times10^6 \times 80 \approx 3.5\times10^9$ positions. Each search calls the network about once per simulation, so playing the games took about $3.5\times10^9\times800 \approx 2.8\times10^{12}$ forward passes.

Training consumed $700{,}000 \times 4{,}096 \approx 2.9\times10^9$ positions, about one pass over the positions played. So generating games took about 1,000 times more network calls than training did. A training step costs about three forward passes (one forward, a backward pass about twice as costly), so in compute the ratio is still about 300 to 1. That is why the hardware was split about 80 to 1 in favor of playing (5,000 TPUs against 64, of different generations).

Self-play at scale is therefore an inference problem. Three things set its cost. The first is throughput, how many positions the hardware can evaluate per second. The second is latency, because the simulations inside one search run one after another. The third is how well leaf evaluations are batched across many games played in parallel. A design review for a self-play system spends most of its time on the game-playing fleet, not the trainer.

The whole loop on Nim

The PyTorch block below runs the whole loop on Nim, from random weights. Nim has only 16 positions up to 15 stones. So each round evaluates the network on all of them at once, and the search reads from that table, which keeps the run to a few seconds. The network reads a one-hot code of the pile size, so it is a lookup table in disguise, and the point is the loop, not generalization.

Watch the policy head learn the winning move even for 13 to 15 stones, beyond the reach of the pure search above. In this run it gets every winning position right by the sixth round, because the value head extends what a 32-simulation search can see.

🔥 In PyTorchfor a machine with PyTorch

The next step removed the rulebook itself. MuZero (Schrittwieser et al. 2020) learned a model of the game's dynamics from experience and searched inside that learned model. Inside its search it never consulted the rules. The real game supplied only the legal moves at the root, the rewards, and the next real position. The same program mastered Go, chess, shogi, and Atari video games. Learned world models carry this idea to environments with no simulator.

When beating your last self goes in circles

Nim, chess, and Go have a property we have used without saying so. There is one best way to play, and a stronger player beats a weaker one. Many games are not like that. In rock-paper-scissors no strategy is best, and "beats" goes around in a circle. Self-play there can beat its previous version forever without getting anywhere. To say what "getting somewhere" means, we need a little game theory.

📐 Matrix game, mixed strategy, best response, Nash equilibrium

A two-player zero-sum matrix game is a payoff matrix $\mathbf{U} \in \R^{m_1\times m_2}$ . The row player picks a row $i$ , the column player picks a column $j$ , and the row player receives $U_{ij}$ while the column player receives $-U_{ij}$ .

A mixed strategy is a probability vector over a player's actions, $\boldsymbol\mu \in \Delta_{m_1}$ for the row player and $\boldsymbol\nu \in \Delta_{m_2}$ for the column player. Here $\Delta_m = \{\mathbf{x} \in \R^m : x_i \ge 0, \sum_i x_i = 1\}$ , the probability simplex, is the set of probability vectors of length $m$ . The row player's expected payoff is $\boldsymbol\mu^\top\mathbf{U}\boldsymbol\nu$ .

A best response to $\boldsymbol\nu$ is any $\boldsymbol\mu$ that maximizes $\boldsymbol\mu^\top\mathbf{U}\boldsymbol\nu$ , and a best response to $\boldsymbol\mu$ is any $\boldsymbol\nu$ that minimizes it. The payoff is linear in each strategy, so some pure strategy, a single action $\mathbf{e}_i$ , is always among the best responses. A Nash equilibrium is a pair $(\boldsymbol\mu^\star, \boldsymbol\nu^\star)$ in which each strategy is a best response to the other.

Here is the standard example. In rock-paper-scissors, with the actions in the order rock, paper, scissors and a payoff of 1 for a win,

\mathbf{U}_{\text{RPS}} = \begin{pmatrix} 0 & -1 & 1 \\ 1 & 0 & -1 \\ -1 & 1 & 0 \end{pmatrix}.

Row 2 is paper. It beats rock in column 1 and loses to scissors in column 3. The only equilibrium is for both players to play each action with probability $\tfrac13$ , and the expected payoff there is 0.

∴ Minimax theorem (von Neumann, 1928)

For every payoff matrix $\mathbf{U}$ ,

\max_{\boldsymbol\mu \in \Delta_{m_1}}\min_{\boldsymbol\nu \in \Delta_{m_2}}\boldsymbol\mu^\top\mathbf{U}\boldsymbol\nu = \min_{\boldsymbol\nu \in \Delta_{m_2}}\max_{\boldsymbol\mu \in \Delta_{m_1}}\boldsymbol\mu^\top\mathbf{U}\boldsymbol\nu = v^\star .

The common number $v^\star$ is the value of the game (von Neumann 1928). A pair is a Nash equilibrium if and only if $\boldsymbol\mu^\star$ attains the outer max on the left and $\boldsymbol\nu^\star$ the outer min on the right. Every equilibrium gives the row player exactly $v^\star$ .

In plain words, with mixed strategies it does not matter who has to commit first. The left side is what the row player can guarantee by announcing a strategy first. The right side is what the column player can hold them to by announcing first. The left can never exceed the right, and that half is easy. Equality is the deep part. The usual proof uses linear programming duality, and below we prove it a second way, from the fact that a kind of self-play converges.

How far is a pair of strategies from equilibrium? The cleanest measure asks how much each player could gain by switching to a best response.

📐 Exploitability (NashConv)

The exploitability of a pair of strategies is what the row player could earn by switching to a best response against $\boldsymbol\nu$ , minus what the column player could hold the row player to by switching to a best response against $\boldsymbol\mu$ ,

\operatorname{expl}(\boldsymbol\mu, \boldsymbol\nu) = \max_{\boldsymbol\mu' \in \Delta_{m_1}}\boldsymbol\mu'^\top\mathbf{U}\boldsymbol\nu - \min_{\boldsymbol\nu' \in \Delta_{m_2}}\boldsymbol\mu^\top\mathbf{U}\boldsymbol\nu' = \max_i\,(\mathbf{U}\boldsymbol\nu)_i - \min_j\,(\boldsymbol\mu^\top\mathbf{U})_j .

To see what it measures, add and subtract the payoff actually received, $\boldsymbol\mu^\top\mathbf{U}\boldsymbol\nu$ ,

\operatorname{expl} = \big[\max_{\boldsymbol\mu'}\boldsymbol\mu'^\top\mathbf{U}\boldsymbol\nu - \boldsymbol\mu^\top\mathbf{U}\boldsymbol\nu\big] + \big[\boldsymbol\mu^\top\mathbf{U}\boldsymbol\nu - \min_{\boldsymbol\nu'}\boldsymbol\mu^\top\mathbf{U}\boldsymbol\nu'\big].

Each bracket is one player's gain from switching, so each is at least 0. Both are 0 exactly when both strategies are best responses.

Now run the most naive self-play, called iterated best response, in which each new version plays the best response to the previous one. Start with rock. The best response to rock is paper, so the next version plays paper. The best response to paper is scissors, and then comes rock again. Each new version beats the previous one every time, so a dashboard that tracks the win rate against the last checkpoint shows 100 percent at every step. Yet the policy goes around a three-step cycle forever. Its exploitability stays at $\operatorname{expl}(\mathbf{e}_R, \mathbf{e}_R) = 2\max(0, 1, -1) = 2$ , the largest possible in this game.

This is the trap to name in an interview. A rising win rate against your last checkpoint is not evidence of progress unless the game is transitive, meaning that if A beats B and B beats C, then A beats C.

✏️ An equilibrium by hand, biased rock-paper-scissors

Make rock's win over scissors pay 2 instead of 1, and cost scissors 2, keeping the game symmetric,

\mathbf{U}_{\text{bias}} = \begin{pmatrix} 0 & -1 & 2 \\ 1 & 0 & -1 \\ -2 & 1 & 0 \end{pmatrix}.

A symmetric game has value 0. Suppose the equilibrium uses all three actions. Then every row must earn the same against $\boldsymbol\nu = (\nu_R, \nu_P, \nu_S)$ , or the row player would drop the worse rows. So $\mathbf{U}\boldsymbol\nu = \mathbf{0}$ , which reads

-\nu_P + 2\nu_S = 0, \qquad \nu_R - \nu_S = 0, \qquad -2\nu_R + \nu_P = 0.

Hence $\nu_R = \nu_S$ and $\nu_P = 2\nu_R$ , and normalizing gives $(\tfrac14, \tfrac12, \tfrac14)$ for both players. The surprise is that making rock more rewarding makes paper the most common move, because paper is what punishes rock.

Iterated best response still cycles from rock to paper to scissors to rock. Its exploitability runs 2, 2, 4, 2, 2, 4, and so on. Scissors against scissors is the worst point, because rock would win 2 against it.

Self-play that converges on average

The fix for cycling is to stop answering only your last self. Fictitious play answers the average of everything the opponent has played so far (Brown 1951, Chapter XXIV). In round $t + 1$ the row player plays a best response to $\bar{\boldsymbol\nu}_t = \frac1t\sum_{s=1}^t\boldsymbol\nu_s$ , and the column player plays a best response to $\bar{\boldsymbol\mu}_t$ . What converges is the pair of average strategies $\bar{\boldsymbol\mu}_T$ and $\bar{\boldsymbol\nu}_T$ , not the strategy played in each round.

The rate can be slow. Empirically the exploitability often falls like $1/\sqrt{T}$ , and with adversarial tie-breaking it can fall far more slowly (Daskalakis and Pan 2014).

A second method has a clean rate and a short proof. It is built on regret, how much better you would have done by playing one fixed action all along. A tiny example comes first.

📐 External regret and regret matching

Suppose the two players play $\boldsymbol\mu_t$ and $\boldsymbol\nu_t$ in rounds $t = 1, \dots, T$ . The row player's average external regret compares what it earned with the best single action in hindsight,

\operatorname{Reg}^{\mu}_T = \max_i \frac1T\sum_{t=1}^T(\mathbf{U}\boldsymbol\nu_t)_i - \frac1T\sum_{t=1}^T\boldsymbol\mu_t^\top\mathbf{U}\boldsymbol\nu_t .

The column player wants the payoff low, so its regret is $\operatorname{Reg}^{\nu}_T = \frac1T\sum_t\boldsymbol\mu_t^\top\mathbf{U}\boldsymbol\nu_t - \min_j\frac1T\sum_t(\boldsymbol\mu_t^\top\mathbf{U})_j$ . An algorithm is no-regret if its average regret goes to 0 or below, whatever the opponent does.

Regret matching (Hart and Mas-Colell 2000) keeps, for each action $i$ , the cumulative regret $R_t(i) = \sum_{s\le t}\big[(\mathbf{U}\boldsymbol\nu_s)_i - \boldsymbol\mu_s^\top\mathbf{U}\boldsymbol\nu_s\big]$ . It plays each action in proportion to the positive part of its regret, $\mu_{t+1}(i) = R_t(i)^+/\sum_k R_t(k)^+$ , or uniformly if no regret is positive. The column player does the same with its own payoff $-\mathbf{U}$ .

In plain words, play more often the actions you wish you had played more often. An action that would have beaten what you actually did gains regret, and the more regret it has piled up, the more often you play it.

The link between regret and equilibrium is an exact identity, not only a bound.

∴ No-regret self-play converges on average

For any sequence of strategies in a two-player zero-sum game,

\operatorname{expl}(\bar{\boldsymbol\mu}_T, \bar{\boldsymbol\nu}_T) = \operatorname{Reg}^{\mu}_T + \operatorname{Reg}^{\nu}_T .

So if both players use no-regret algorithms, the exploitability of their average strategies goes to 0. Regret matching guarantees $\operatorname{Reg}_T \le D\sqrt{m/T}$ for a player with $m$ actions, where $D = \max_{ij}U_{ij} - \min_{ij}U_{ij}$ is the range of the payoffs. So the averages are within $D(\sqrt{m_1} + \sqrt{m_2})/\sqrt{T}$ of equilibrium in exploitability.

∂ Proof of the identity, the regret-matching bound, and the minimax theorem

The identity. The payoff is linear in each strategy, so $\frac1T\sum_t(\mathbf{U}\boldsymbol\nu_t)_i = (\mathbf{U}\bar{\boldsymbol\nu}_T)_i$ and $\frac1T\sum_t(\boldsymbol\mu_t^\top\mathbf{U})_j = (\bar{\boldsymbol\mu}_T^\top\mathbf{U})_j$ . Substituting into the definitions,

\operatorname{Reg}^{\mu}_T + \operatorname{Reg}^{\nu}_T = \max_i(\mathbf{U}\bar{\boldsymbol\nu}_T)_i - \underbrace{\tfrac1T\textstyle\sum_t\boldsymbol\mu_t^\top\mathbf{U}\boldsymbol\nu_t}_{\text{cancels}} + \tfrac1T\textstyle\sum_t\boldsymbol\mu_t^\top\mathbf{U}\boldsymbol\nu_t - \min_j(\bar{\boldsymbol\mu}_T^\top\mathbf{U})_j ,

and what is left is the definition of $\operatorname{expl}(\bar{\boldsymbol\mu}_T, \bar{\boldsymbol\nu}_T)$ . The average payoff actually received cancels because one player's gain is the other's loss.

The regret-matching bound. Take the row player, and let $\mathbf{r}_t \in \R^{m}$ be its instantaneous regret vector, $r_t(i) = (\mathbf{U}\boldsymbol\nu_t)_i - \boldsymbol\mu_t^\top\mathbf{U}\boldsymbol\nu_t$ , so $\mathbf{R}_t = \mathbf{R}_{t-1} + \mathbf{r}_t$ . Two facts do all the work. First, $\mathbf{R}_{t-1}^+\cdot\mathbf{r}_t = 0$ . Indeed $\boldsymbol\mu_t = \mathbf{R}_{t-1}^+/\lVert\mathbf{R}_{t-1}^+\rVert_1$ , and with $\mathbf{u} = \mathbf{U}\boldsymbol\nu_t$ ,

\mathbf{R}_{t-1}^+\cdot\mathbf{r}_t = \mathbf{R}_{t-1}^+\cdot\mathbf{u} - \lVert\mathbf{R}_{t-1}^+\rVert_1\,\boldsymbol\mu_t\cdot\mathbf{u} = \mathbf{R}_{t-1}^+\cdot\mathbf{u} - \mathbf{R}_{t-1}^+\cdot\mathbf{u} = 0

(if $\mathbf{R}_{t-1}^+ = \mathbf{0}$ the product is 0 at once). Second, taking the positive part never increases distance. Coordinate by coordinate, $\big((x + y)^+\big)^2 \le (x^+ + y)^2$ , which you can check in the two cases $x \ge 0$ and $x$ negative. Hence

\lVert\mathbf{R}_t^+\rVert^2 \le \lVert\mathbf{R}_{t-1}^+ + \mathbf{r}_t\rVert^2 = \lVert\mathbf{R}_{t-1}^+\rVert^2 + 2\,\mathbf{R}_{t-1}^+\cdot\mathbf{r}_t + \lVert\mathbf{r}_t\rVert^2 \le \lVert\mathbf{R}_{t-1}^+\rVert^2 + mD^2 ,

since each $|r_t(i)| \le D$ . Summing from $t = 1$ to $T$ gives $\lVert\mathbf{R}_T^+\rVert^2 \le TmD^2$ . The largest regret is at most the norm of the positive part, so $\max_i R_T(i) \le D\sqrt{mT}$ , and dividing by $T$ gives $\operatorname{Reg}^{\mu}_T \le D\sqrt{m/T}$ .

The minimax theorem. Weak duality, $\max_{\boldsymbol\mu}\min_{\boldsymbol\nu} \le \min_{\boldsymbol\nu}\max_{\boldsymbol\mu}$ , always holds. For the reverse, run regret matching for both players for $T$ rounds and let $\epsilon_T = \operatorname{expl}(\bar{\boldsymbol\mu}_T, \bar{\boldsymbol\nu}_T)$ , which is at most $D(\sqrt{m_1} + \sqrt{m_2})/\sqrt T$ by the two results above. Then

\min_{\boldsymbol\nu}\max_{\boldsymbol\mu}\boldsymbol\mu^\top\mathbf{U}\boldsymbol\nu \le \max_{\boldsymbol\mu}\boldsymbol\mu^\top\mathbf{U}\bar{\boldsymbol\nu}_T = \min_{\boldsymbol\nu}\bar{\boldsymbol\mu}_T^\top\mathbf{U}\boldsymbol\nu + \epsilon_T \le \max_{\boldsymbol\mu}\min_{\boldsymbol\nu}\boldsymbol\mu^\top\mathbf{U}\boldsymbol\nu + \epsilon_T ,

where the middle equality is the definition of exploitability. Letting $T \to \infty$ sends $\epsilon_T \to 0$ and gives the reverse inequality. ∎

The theorem teaches two things. First, the averages converge, but nothing says the strategies played in each round do, and in rock-paper-scissors they keep circling. Second, the identity makes exploitability something you can compute during training from logged regrets, without ever solving the game. Regret matching extended to games with many decision points and hidden information is called counterfactual regret minimization (Zinkevich et al. 2007). It is the engine inside the poker programs that beat professionals.

The cell runs all three kinds of self-play from rock against rock and plots the exploitability of the average strategy.

Python · cell 4

Look at iterated best response first. In plain rock-paper-scissors its cycle visits each action equally often. So its average lands exactly on the uniform equilibrium every third round, and its line drops to the bottom of the plot. That is luck, not convergence. In the biased game the cycle still spends a third of its time on each action. Its average goes to $(\tfrac13, \tfrac13, \tfrac13)$ , which is not the equilibrium, and its exploitability stays near $\tfrac23$ .

Fictitious play and regret matching both fall roughly like $1/\sqrt{T}$ in both games, and both find the biased game's $(\tfrac14, \tfrac12, \tfrac14)$ . The last strategy played by every method is still exploitable. The widget shows the same dynamics on the triangle of mixed strategies, where the current strategy circles while the average spirals in.

Core

Self-play that cycles and self-play that converges

Write three functions for a zero-sum matrix game. The payoff U has shape (m1, m2) and is the row player's payoff.

exploitability(U, mu, nu) returns $\max_i(\mathbf{U}\boldsymbol\nu)_i - \min_j(\boldsymbol\mu^\top\mathbf{U})_j$ , the sum of both players' gains from switching. It is 2 for rock against rock.
best_response_dynamics(U, T) is iterated best response. Both players start at action 0. In each of the next $T - 1$ rounds, both switch at once to a pure best response to the other's previous strategy. Return (mus, nus), the strategies of every round including the first, with shapes (T, m1) and (T, m2).
fictitious_play(U, T) has both players play action 0 in round 1. In each later round, each player plays a pure best response to the average of the opponent's past actions, both at once. Return the average strategies (mu_bar, nu_bar) over all $T$ rounds.

Break every tie toward the lowest index (np.argmax, np.argmin).

import numpy as np

def exploitability(U, mu, nu):
    return float((U @ nu).max() - (mu @ U).max())

def best_response_dynamics(U, T):
    raise NotImplementedError

Tests, hints, verified solutions, and AI help, with Pro at $7.99 a month.

Research

Regret matching and the regret-to-Nash identity

Write regret_matching(U, T), which runs regret matching for both players of a zero-sum matrix game. Return (mu_bar, nu_bar, reg_mu, reg_nu), the average strategies over the $T$ rounds and the two average external regrets $\operatorname{Reg}^\mu_T$ and $\operatorname{Reg}^\nu_T$ defined above. Each regret is the largest average regret over actions, and it may be negative.

In round 1 both players play action 0 as a pure strategy.
After each round, each player adds its instantaneous regret vector to its cumulative regrets, using expected payoffs against the opponent's mixed strategy. The row player's vector is $(\mathbf{U}\boldsymbol\nu_t)_i - \boldsymbol\mu_t^\top\mathbf{U}\boldsymbol\nu_t$ , and the column player's is $-(\boldsymbol\mu_t^\top\mathbf{U})_j + \boldsymbol\mu_t^\top\mathbf{U}\boldsymbol\nu_t$ .
Each player then plays its positive regrets normalized to sum to 1, or uniform if none is positive.

The tests check the identity $\operatorname{expl}(\bar{\boldsymbol\mu}_T, \bar{\boldsymbol\nu}_T) = \operatorname{Reg}^\mu_T + \operatorname{Reg}^\nu_T$ and the bound $D\sqrt{m/T}$ .

import numpy as np

def regret_matching(U, T):
    raise NotImplementedError

Tests, hints, verified solutions, and AI help, with Pro at $7.99 a month.

Populations of past selves stand in for the average

Fictitious play needs the average of all past strategies. That is easy for a vector of three numbers and hard for a neural network, because averaging the weights of two networks does not average how they play. Large systems keep a population of past networks instead and play mixtures of them.

Lanctot et al. (2017) put the options into one algorithm, Policy-Space Response Oracles (PSRO). It repeats three steps.

Play every pair of policies in the population against each other. The results form a payoff table, a small matrix game whose actions are whole policies, called the meta-game.
Choose a mixture over the population with a rule called the meta-solver.
Train a new policy by reinforcement learning to beat that mixture, and add it to the population.

The meta-solver decides which classical method you get. All weight on the newest policy gives iterated best response. A uniform mixture gives fictitious play. The meta-game's Nash equilibrium gives the double oracle method (McMahan, Gordon, and Blum 2003).

In a finite game double oracle stops after finitely many steps. Each step either adds a strategy not yet in the population, or finds that the small game's equilibrium is already an equilibrium of the full game. In rock-paper-scissors, starting from rock, it adds paper (the best response to rock) and then scissors (the best response to paper, which is the equilibrium of the rock-paper game). The equilibrium of all three is the uniform mixture. Every action earns the same against it, so there is nothing left to add.

AlphaStar (Vinyals et al. 2019) built the largest league of this kind for a game, the real-time strategy game StarCraft II. It began from networks trained on human replays, so it did not start from scratch, and then trained three kinds of agents.

Main agents train against themselves and the whole league, meeting past opponents more often the more they lose to them.
Main exploiters train only against the current main agents, to expose their weaknesses.
League exploiters train against the whole league, to find strategies that no one handles.

Frozen copies join the league over time, so every weakness found stays in the pool of opponents.

In practice, a population is how you get the benefit of fictitious play with neural networks. The cost is a growing payoff table and many training runs in parallel, and the reward is a policy that does not lose to strategies it has already met.

🔬 Why one self-play run overfits to itself, and where cycles live

Lanctot et al. also measured why populations matter. Agents trained together by independent reinforcement learning overfit to each other's habits. Paired with agents from another training run, they did worse, sometimes much worse. The authors called this joint-policy correlation. A single self-play run learns to beat, or to work with, one opponent and nothing more.

Real games mix two parts. In the transitive part some strategies really are better. In the cyclic part who beats whom depends on style. Czarnecki et al. (2020) argued that many real games look like a spinning top, with a wide cyclic region in the middle of the skill range that a population must cover before progress continues. That is one explanation for why leagues help most at intermediate strength.

Superhuman is not unexploitable

AlphaZero-style Go programs became so strong that people treated them as nearly perfect. In 2022 and 2023 a research group showed that they are not.

📌 Adversarial policies beat superhuman Go programs

Wang et al. (2023) trained an adversarial policy, a player trained by reinforcement learning to beat one fixed opponent. Its target was a frozen copy of KataGo, the strongest open-source Go program. It won more than 97 percent of games against KataGo at superhuman settings. The adversary plays badly by human standards, and human amateurs beat it. So the amateur beats the adversary, the adversary beats KataGo, and KataGo beats the amateur, a rock-paper-scissors cycle among Go players.

The adversary wins by steering KataGo into unusual positions, built around a circular group of stones, which KataGo misjudges and then blunders in. Human experts learned the trick and used it to beat superhuman programs without computer help. The attack transferred to other superhuman Go programs. And the core weakness persisted even in KataGo versions trained to defend against it.

The lesson for anyone evaluating a self-play system is that two numbers answer two different questions. Elo against past versions and against humans measures the transitive part of strength, which can be enormous. Exploitability asks how well the best possible counter-strategy does. Self-play covers the positions that self-play visits, and an optimizer aimed squarely at the system finds the ones it never visits. Computing a best response in a large game is as hard as training, so in practice exploitability is estimated by training exploiters, which is what AlphaStar's league builds in.

The same lesson applies to every learned judge in this book. A reward model is a network trained to score answers the way people would. A policy trained by reinforcement learning against a frozen reward model is an adversarial policy against that model. The result is reward hacking, answers that score high and are not good (§16.5). It is also the central risk of the loops in §26.4, where a model judges its own outputs.

🔬 Beyond two-player zero-sum games

In a two-player zero-sum game, an equilibrium strategy is safe to play against anyone, because the minimax theorem guarantees the value whatever the opponent does. With more players, or with shared interests, that guarantee disappears. Strategies learned in self-play can be excellent with copies of themselves and useless with people, because they rely on conventions the copies share. In the cooperative card game Hanabi, independently trained self-play agents invent arbitrary signaling conventions and fail when paired with partners from other runs (Hu et al. 2020).

Diplomacy is a seven-player game of negotiation and betrayal. Meta's CICERO (Meta FAIR Diplomacy Team 2022) paired a dialogue model with a planner. A key ingredient of the planner was a search regularized toward human play, called piKL (Jacob et al. 2022). Instead of picking the move with the highest value, it maximizes value minus a KL penalty toward a policy $\pi_{\text{human}}$ imitated from human games, $\mathbb{E}_{a\sim\pi}[Q(s,a)] - \beta\,D_{\mathrm{KL}}(\pi\,\Vert\,\pi_{\text{human}})$ . The maximizer is

\pi(a\mid s) \propto \pi_{\text{human}}(a\mid s)\exp\!\big(Q(s,a)/\beta\big),

the KL-regularized optimum of §17.1, with the human policy as the reference and the search's action values in place of the reward. A small $\beta$ trusts the search, a large $\beta$ stays human-like, and values in between play strongly while staying predictable to human partners. In Diplomacy every player's policy changes at once, so piKL runs inside a regret-minimization loop rather than as a single search. CICERO played 40 games in an anonymous online Diplomacy league. It scored more than double the average human player and ranked in the top 10 percent of participants who played more than one game.

Language models have the opponent but not the referee

Back to the opening question. Why does self-play make superhuman Go players but not superhuman essay writers? Go supplies three things, and open-ended language tasks lack the most important one.

An exact reward. The rules of Go say who won. Nothing says which of two essays is better except a person, or a model trained to imitate people.
An opponent of matching strength. This one transfers, since a model can always play a copy of itself.
Search with a reliable value. Tree search over text is possible (§18.6), but the value of a half-written argument is far harder to learn than the value of a Go position.

Every self-play method for language models is a way to supply the first ingredient. Some play real games whose rules decide the winner. Some make the training data the referee. Some use a judge or a reward model. And some invent problems whose answers a program can check. The table sorts the main methods by what decides the winner, and that column predicts how far each can go.

method	the game	what decides the winner	what limits it
SPAG (Cheng et al. 2024)	an attacker tries to make a defender say a secret word	the game's rules	whether game skill transfers to reasoning, an empirical question
SPIRAL (Liu et al. 2025)	tic-tac-toe, Kuhn poker, and a simple negotiation, against itself	the game's rules	the games' depth and transfer
SPIN (Chen et al. 2024)	tell the dataset's answers from the model's own	the supervised data, implicitly	the data distribution, a theorem
Debate training (Arnesen et al. 2024)	two copies argue for opposite answers before a judge	a fine-tuned GPT-4-Turbo judge	the judge's accuracy
Language Self-Play (Kuba et al. 2025)	a challenger writes queries, a solver answers	a reward model plus a self-judged quality score	the reward model, which can be hacked
Absolute Zero (Zhao et al. 2025)	one model proposes and solves code tasks	a Python executor	tasks a program can check
R-Zero (Huang et al. 2025)	a challenger writes math questions, a solver answers	the solver's own majority vote	label accuracy, which falls as questions get harder
Self-play SWE-RL (Wei et al. 2025)	inject a bug into a real repository, then repair it	tests the bug injector writes, run in the repository	how well tests pin down the bugs
AlphaProof (Hubert et al. 2025)	prove automatically formalized statements	the Lean proof checker	what can be formalized

Games played in language

The most direct approach keeps a real game and plays it in words, with the rules as the referee. The rewards are then exact, but they reward skill at the game, and how much of that skill carries over to math and code is a question only experiments can answer. The box gives the two main results.

🔬 Games played in language

In Adversarial Taboo, an attacker and a defender, two copies of one model, talk around a target word that only the attacker can see. The attacker wins if the defender says the word without realizing it, and the defender wins by inferring the word and naming it. Training both roles by reinforcement learning on game outcomes improved the model on a broad range of reasoning benchmarks, and repeating the self-play kept improving it (Cheng et al. 2024).

Liu et al. (2025) ran fully online, multi-turn self-play on tic-tac-toe, Kuhn poker (a three-card poker game), and a simple negotiation game. To keep the two sides' learning stable they used a separate reward baseline for each player role. Across four models, the average over eight reasoning benchmarks rose by up to about 10 percentage points (Qwen3-4B-Base went from 34.0 to 44.5 percent). That beat supervised fine-tuning on 25,000 expert game trajectories.

Self-play against your previous self

Self-play fine-tuning (SPIN, Chen et al. 2024) plays a game whose referee is a supervised dataset of prompts and target answers. The targets may be written by people or, as in the paper's experiments, by a stronger model. Call them the data answers. SPIN starts from a model already fine-tuned on that data, at the point where more supervised training no longer helps.

Each iteration plays one round of a game. The previous model writes its own answer to each prompt. The new model is trained to tell the data answer from the previous model's answer. The previous model is the opponent, trying to make the two indistinguishable.

One trick turns this game into a single loss. Score each answer with the new model itself, by how much more likely the new model finds it than the previous model did. Raising the score of data answers and lowering the score of the model's own answers is then a preference loss. It is the loss of direct preference optimization (DPO), which trains on pairs of a preferred and a rejected answer and raises the first's log-probability relative to a fixed reference model while lowering the second's (§17.2 derived it).

📐 The SPIN objective

Let $q$ be the prompt distribution, $p_{\text{data}}(\cdot\mid x)$ the distribution of supervised answers to a prompt $x$ , $\pi_{\theta_t}$ the current model, $\lambda_{\text{spin}} > 0$ a regularization parameter, and $\ell(u) = \log(1 + e^{-u})$ the logistic loss. Iteration $t$ minimizes

\mathcal{L}_{\text{SPIN}}(\theta;\theta_t) = \mathbb{E}_{x\sim q,\;y\sim p_{\text{data}}(\cdot\mid x),\;y'\sim\pi_{\theta_t}(\cdot\mid x)}\Big[\ell\Big(\lambda_{\text{spin}}\log\frac{\pi_\theta(y\mid x)}{\pi_{\theta_t}(y\mid x)} - \lambda_{\text{spin}}\log\frac{\pi_\theta(y'\mid x)}{\pi_{\theta_t}(y'\mid x)}\Big)\Big]

over $\theta$ , and sets $\theta_{t+1}$ to the minimizer.

In plain words, this is the DPO loss with the data answer $y$ always chosen, the model's own answer $y'$ always rejected, and the previous model as the reference. No preference labels are needed, only the supervised data.

What does each iteration do to the model? The exact answer fits in one line, and a tiny example shows it first.

✏️ One SPIN iteration on a prompt with four answers

Take one prompt with four possible answers. The data puts probabilities $(0.5, 0.3, 0.15, 0.05)$ on them and the current model $(0.1, 0.2, 0.3, 0.4)$ . With $\lambda_{\text{spin}} = 2$ , the theorem below says the best possible next model is the normalized geometric mean of the two. The products are $(0.05, 0.06, 0.045, 0.02)$ , their square roots are $(0.224, 0.245, 0.212, 0.141)$ , and normalizing gives $(0.27, 0.30, 0.26, 0.17)$ .

In log space the new model lands exactly halfway from the model to the data. With $\lambda_{\text{spin}} = 1$ it would land on the data in one step. In no case does it move past the data.

The geometric mean in the example is not a coincidence of the numbers. The theorem below gives the update for any data and any $\lambda_{\text{spin}}$ , shows that the data distribution is where the iterations stop, and says for which $\lambda_{\text{spin}}$ they get there.

∴ SPIN's fixed point and its exact update

(Chen et al. 2024, Theorems 5.2 and 5.4.) Assume $p_{\text{data}}$ and $\pi_{\theta_t}$ give every answer positive probability and the model class can represent the distributions below.

(a) If $\pi_{\theta_t} = p_{\text{data}}$ , then $\theta_t$ minimizes $\mathcal{L}_{\text{SPIN}}(\cdot\,;\theta_t)$ for every $\lambda_{\text{spin}}$ , so the data distribution is a fixed point.

(b) With the logistic loss, the minimizer satisfies

\pi_{\theta_{t+1}}(y\mid x) \propto \pi_{\theta_t}(y\mid x)\Big(\frac{p_{\text{data}}(y\mid x)}{\pi_{\theta_t}(y\mid x)}\Big)^{1/\lambda_{\text{spin}}} .

(c) This part follows from (b) and is derived below, not taken from the paper. The log-ratio errors $e_t(y) = \log\pi_{\theta_t}(y\mid x) - \log p_{\text{data}}(y\mid x)$ obey $e_{t+1} = (1 - 1/\lambda_{\text{spin}})\,e_t + C_t$ for a constant $C_t$ . From every starting point $\pi_{\theta_0} \ne p_{\text{data}}$ , $\pi_{\theta_t} \to p_{\text{data}}$ if and only if $\lambda_{\text{spin}} > \tfrac12$ . The convergence takes one step at $\lambda_{\text{spin}} = 1$ , is monotone for $\lambda_{\text{spin}} > 1$ , and oscillates but converges for $\tfrac12 < \lambda_{\text{spin}} < 1$ . At $\lambda_{\text{spin}} = \tfrac12$ the model flips back and forth between two distributions, and below that the oscillation grows.

∂ Proof of the update and of the convergence condition

Fix a prompt and drop $x$ . Write $\pi_t = \pi_{\theta_t}$ and $g(y) = \lambda_{\text{spin}}\log\big(\pi(y)/\pi_t(y)\big)$ for the candidate $\pi$ . The loss is $\sum_{a,b}p_{\text{data}}(a)\,\pi_t(b)\,\ell\big(g(a) - g(b)\big)$ . Group each pair $a \ne b$ with its mirror image $b, a$ , and let $w_{ab} = p_{\text{data}}(a)\pi_t(b) > 0$ . The pair contributes $h(d) = w_{ab}\log(1 + e^{-d}) + w_{ba}\log(1 + e^{d})$ with $d = g(a) - g(b)$ . This is strictly convex in $d$ , and $h'(d) = -w_{ab}\,\sigma(-d) + w_{ba}\,\sigma(d)$ , which vanishes when $\sigma(d) = w_{ab}/(w_{ab} + w_{ba})$ , that is at $d^\star = \log(w_{ab}/w_{ba})$ . The diagonal terms $a = b$ contribute the constant $\ell(0)$ .

Now take $g(y) = \log\big(p_{\text{data}}(y)/\pi_t(y)\big) + C$ for any constant $C$ . Then for every pair $g(a) - g(b) = \log\frac{p_{\text{data}}(a)\pi_t(b)}{p_{\text{data}}(b)\pi_t(a)} = d^\star$ , so this single $g$ minimizes every term at once and therefore the sum. By strict convexity any minimizer must give every pair the same differences, so the minimizing $g$ is unique up to $C$ . Solving $\lambda_{\text{spin}}\log(\pi/\pi_t) = \log(p_{\text{data}}/\pi_t) + C$ gives $\pi = \pi_t\,(p_{\text{data}}/\pi_t)^{1/\lambda_{\text{spin}}}\,e^{C/\lambda_{\text{spin}}}$ , and normalization fixes $C$ . This is (b). If $\pi_t = p_{\text{data}}$ , the update returns $\pi_t$ , which is (a) for the logistic loss (the paper proves it for every convex decreasing $\ell$ ).

Taking logs of (b), $\log\pi_{t+1} = (1 - 1/\lambda_{\text{spin}})\log\pi_t + (1/\lambda_{\text{spin}})\log p_{\text{data}} + C_t$ . Subtract $\log p_{\text{data}}$ from both sides to get $e_{t+1} = (1 - 1/\lambda_{\text{spin}})e_t + C_t$ . The constant cannot change the shape of the distribution, so measure the error after removing it, $\tilde e_t = e_t - \operatorname{mean}_y e_t$ , which obeys $\tilde e_{t+1} = (1 - 1/\lambda_{\text{spin}})\tilde e_t$ exactly. If $\tilde e_t = 0$ , then $\pi_t = p_{\text{data}}e^{c}$ for a constant $c$ , and since both sum to 1, $c = 0$ . So $\pi_t \to p_{\text{data}}$ exactly when $\lvert 1 - 1/\lambda_{\text{spin}}\rvert < 1$ , which for $\lambda_{\text{spin}} > 0$ means $\lambda_{\text{spin}} > \tfrac12$ . The factor is 0 at $\lambda_{\text{spin}} = 1$ , in $(0, 1)$ above 1, in $(-1, 0)$ between $\tfrac12$ and 1, exactly $-1$ at $\tfrac12$ , and below $-1$ under $\tfrac12$ . ∎

The cell runs the exact update from the tiny example for several values of $\lambda_{\text{spin}}$ . It then checks part (b) against the loss itself.

Python · cell 5

The theorem explains SPIN's results and its limit at the same time. The fixed point is $p_{\text{data}}$ , so SPIN cannot exceed the quality of its supervised data. It can only pull a model that has not fully absorbed the data closer to it. In practice that makes SPIN a way to squeeze more out of a supervised dataset after ordinary fine-tuning has stalled, and never a way past it.

📌 SPIN at 7 billion parameters

Chen et al. started from zephyr-7b-sft-full, a Mistral-7B model already fine-tuned on the Ultrachat200k dialogue dataset, whose dialogues were generated with OpenAI's chat models. They used 50,000 of its prompts. The score is the average over the six benchmarks of the Open LLM Leaderboard, a public suite of knowledge, reasoning, and math tests. Four SPIN iterations raised it from 58.14 to 60.80, 62.12, 62.97, and 63.16, gains that shrink the way a contraction toward a limit would. One more epoch of ordinary supervised training instead lowered it to 57.23. The second iteration already beat zephyr-7b-dpo-full (61.31), which had been trained with 62,000 GPT-4 preference labels. The paper itself names the limit, that the target data distribution puts a ceiling on the fine-tuned model.

⚠️ The theorem gives the target, not the step

The exact update is an idealization, and the gap matters. The paper's experiments set $\lambda_{\text{spin}}$ (called $\beta$ in its appendix) to 0.1 in the first three iterations. For that value the exact update would multiply the centered log errors by $-9$ and diverge. The actual runs take two epochs of small gradient steps per iteration, with a peak learning rate of $5\times10^{-7}$ , lowered to $10^{-7}$ in the last two iterations. These steps move only part of the way toward the minimizer. In the last iteration, close to convergence, the authors raised $\lambda_{\text{spin}}$ to 5.0 for stability, which by part (c) is a gentle contraction. Read the theorem as saying where each iteration points, not how far it travels.

Core

SPIN in a table

Simulate SPIN exactly on a finite set of $K$ answers.

spin_tabular(p0, p_data, lam, iters) applies the exact update $p_{t+1} \propto p_t\,(p_{\text{data}}/p_t)^{1/\lambda}$ for iters iterations. Return the whole trajectory as an array of shape (iters + 1, K), starting with p0.
Work in log space and subtract the maximum before exponentiating, so that tiny probabilities and small $\lambda$ cannot overflow. Carry the log-probabilities from one iteration to the next instead of taking the log of the previous row again. Below $\lambda = \tfrac12$ some entries underflow to exactly 0, and the log of 0 would turn the next row into nan.
spin_rate(lam) returns the factor $\lvert 1 - 1/\lambda\rvert$ by which the centered log-ratio error shrinks or grows each iteration.

import numpy as np

def spin_tabular(p0, p_data, lam, iters):
    traj = [np.asarray(p0, float)]
    for _ in range(iters):
        p = traj[-1]
        q = p * (p_data / p) ** (1 / lam)
        traj.append(q / q.sum())

Tests, hints, verified solutions, and AI help, with Pro at $7.99 a month.

Debate, and games with a learned referee

§19.5 introduced debate as a way to supervise answers a judge cannot check directly. Two models argue for different answers, and a weaker judge decides who won. The hope is that the truth is easier to defend than a lie. Training the debaters to win is self-play with the judge's verdict as the reward. This is not the cooperative debate of §24.7, where several agents revise toward one answer and a vote decides. Here the two sides argue against each other and a third party judges.

Arnesen, Rein, and Michael (2024) trained Llama-3-8B debaters this way on QuALITY-HARD, hard multiple-choice questions about long stories. The judge was a fine-tuned GPT-4-Turbo that saw only the debate, not the story. The judge was 4 percentage points more accurate on debates between the fully trained debaters than on debates between the starting ones ( $p < 10^{-6}$ ).

The authors also trained consultancy, where one model argues for an assigned answer with no opponent, and found no such gain. Across checkpoints the judge was right 77 percent of the time on debates and 68 percent on single consultancies.

Why does the opponent help? The authors checked the obvious story, that each debater refutes the other's misleading claims. It explained little. A judge shown two separate consultancies side by side, one for each answer, was right 75 percent of the time, close to debate's 77, though neither side ever answered the other. Most of the gain came from the judge seeing the best evidence for both answers. The trained consultant, by contrast, quoted less evidence and repeated itself more. Its habits persuaded the judge it was trained against far more than they persuaded a different, untrained judge model.

Language Self-Play (Kuba et al. 2025) shows the other side of a learned referee. One model plays two roles. The challenger writes queries meant to be hard, and the solver answers them. A reward model scores the answers, and a KL penalty toward the starting model keeps the challenger's queries meaningful. The full method adds a quality score that the starting model, prompted as a judge, gives each query and answer.

The authors started from Llama-3.2-3B-Instruct and used no training data. They measured win rates on AlpacaEval, a standard set of instruction prompts, against the starting model, with Llama-4-Maverick as the judge. On this scale the starting model itself scores 26.5 percent. The full method reached 36.4 percent and the version without the quality score 32.0 percent, while GRPO trained on real instruction data reached 38.8 percent.

The quality score fixes a failure the authors saw early on. With a different reward model in their first experiments, the game degenerated. The solver learned to answer most queries in Python, which that reward model liked and people do not. Self-play against a learned referee is an optimizer aimed at the referee.

Curricula that write their own problems

An opponent can also be a teacher that writes the solver's problems. One role proposes tasks and another solves them. Reward the proposer for tasks that are hard but not impossible for the solver. Then the pair builds its own curriculum, a sequence of tasks whose difficulty rises with the solver's skill.

📜 Proposers and solvers before language models

Sukhbaatar et al. (2018) set this up for reinforcement learning agents. One agent, Alice, performs a sequence of actions and stops, and a second agent, Bob, must undo or repeat them. Let $t_A$ and $t_B$ be the times they take and $\gamma > 0$ a reward scale. Bob's reward is $-\gamma t_B$ , and Alice's is $\gamma\max(0, t_B - t_A)$ . Alice is paid only when Bob needs longer than she did, so she proposes tasks just beyond Bob's reach that she has just shown are possible. In OpenAI's hide-and-seek (Baker et al. 2019), the same pressure between two teams produced six distinct phases of strategy, including building shelters from boxes and using ramps to break into them.

With language models, the two roles are played by one network (Absolute Zero) or by two copies trained from the same base model (R-Zero). The question is always what makes a proposed task worth solving.

📐 Learnability rewards for a proposer

For a proposed task $x$ , let the solver make $G$ attempts with outcomes $R_1, \dots, R_G \in \{0, 1\}$ , and let $\bar r_{\text{solve}} = \frac1G\sum_g R_g$ be its success rate. It estimates the probability $p(x)$ that one attempt succeeds (§18.2). Absolute Zero (Zhao et al. 2025) rewards the proposer with

r_{\text{prop}}(x) = \begin{cases} 0 & \text{if } \bar r_{\text{solve}} = 0, \\ 1 - \bar r_{\text{solve}} & \text{otherwise.} \end{cases}

R-Zero (Huang et al. 2025) has no answer checker. It takes the most common of the solver's $m$ answers as a stand-in for the correct one, called a pseudo-label. It rewards the challenger with $1 - 2\lvert\hat p - \tfrac12\rvert$ , where $\hat p$ is the fraction of the $m$ answers that agree with the pseudo-label.

Why tasks of middling difficulty? Because that is where learning happens. Recall GRPO, the reinforcement learning method of §18.4. It samples a group of attempts at each task and scores each attempt by how much better it did than the group's average, a score called its advantage.

The general statement has three parts, all from earlier chapters. For a task solved with probability $p$ , the reward $R \in \{0, 1\}$ has variance $p(1 - p)$ . If the solver's log-odds of success on the task are a parameter $\phi$ , so that $p = \sigma(\phi)$ , then the gradient of success is $\mathrm{d}p/\mathrm{d}\phi = p(1 - p)$ . And a group of $G$ attempts is mixed with probability $1 - p^G - (1 - p)^G$ . All three vanish at $p = 0$ and $p = 1$ and are largest in between.

The two rewards aim at different points of that range. R-Zero's reward peaks at exactly $\hat p = \tfrac12$ , like $p(1-p)$ . Absolute Zero's reward peaks at the hardest task that is solved at least once, $\bar r_{\text{solve}} = 1/G$ , a deliberate push toward the edge of what the solver can do. The cell draws all four curves.

Python · cell 6

✏️ Self-proposed curricula at real scale

Absolute Zero uses a code executor both to check that proposed tasks are valid and to check answers. A task is a triple of a program, an input, and an output, with one part hidden. That gives three kinds of reasoning, deduction (predict the output), abduction (find an input that produces the output), and induction (write the program from examples). Trained with no external data, the 7B base and coder models raised their math averages by 10.9 and 15.2 points. Expert code models trained by ordinary reinforcement learning with verifiable rewards on code, where a program checks each answer (§18.3), gained only 0.65 points in math. The authors also reported an "uh-oh moment", a concerning chain of thought from a Llama-3.1-8B run, a reminder that self-directed training still needs oversight.

R-Zero keeps a question for the solver only if 3 to 7 of the solver's 10 answers agree with the pseudo-label. After three iterations Qwen3-4B-Base gained 6.49 points on math benchmarks and 7.54 on general reasoning. But measured against GPT-4o's answers, the pseudo-labels' accuracy fell from 79.0 percent to 63.0 percent by the third iteration as the questions got harder. Every model size eventually got worse, and the smallest (0.6B) peaked after the first iteration. The authors found that falling label accuracy is not the whole cause of this collapse, since no single accuracy level triggered it across model sizes. But the trend is clear. A referee built from the model's own votes is least reliable exactly where the curriculum pushes.

Self-play SWE-RL (Wei et al. 2025) needs only real code repositories with their dependencies installed. One agent injects a bug, specified by a test patch rather than a written issue, and checks by running it that the bug is real. The same agent then repairs it, and the bug's tests grade the repair. It improved by 10.4 points on SWE-bench Verified and 7.8 on SWE-Bench Pro, two benchmarks of real software issues graded by tests. It stayed above a baseline trained on human-written issues throughout training, even though the benchmarks pose natural-language issues it never saw in self-play.

AlphaProof (Hubert et al. 2025) is the closest heir of AlphaZero. It learns to write formal proofs in Lean, a language whose checker verifies every step of a proof. It trains by reinforcement learning on millions of problems translated automatically into Lean. For the hardest problems it also learns at test time, generating and training on millions of variants of the problem at hand. At the 2024 International Mathematical Olympiad it solved three of the five non-geometry problems, including the hardest. Together with AlphaGeometry 2, after multi-day computation, it reached the score of a silver medalist. Lean is the rulebook, so every accepted proof is a correct proof of its formal statement. The remaining risk is a statement that was formalized wrongly.

The pattern across the four is the chapter's argument again. The loops grounded in a program (an executor, tests, a proof checker) kept improving, and the loop grounded in its own votes degraded as it pushed. When you design a proposer-solver loop, the first question is what checks the answers, and the second is whether that check stays reliable as the tasks get harder.

Core

Learnability rewards for a proposer

Write two functions that score proposed tasks.

proposer_rewards(outcomes) takes an integer array of shape (n_tasks, G) of 0 or 1 solver results. It returns four arrays of shape (n_tasks,), in this order. They are the success rates $\bar r_{\text{solve}}$ , Absolute Zero's reward $r_{\text{prop}}$ , the uncertainty reward $1 - 2\lvert\bar r - \tfrac12\rvert$ , and the reward variance $\bar r(1 - \bar r)$ .
majority_agreement(answers) takes an integer array of shape (n_tasks, m) of the solver's final answers, given as answer ids. It returns the pseudo-labels (the most frequent answer per task, with ties going to the smallest id) and the agreement rates $\hat p$ (the fraction of answers equal to the pseudo-label).

import numpy as np

def proposer_rewards(outcomes):
    r = outcomes.sum(axis=1)
    azr = 1 - r
    rzero = 1 - 2 * (r - 0.5)
    return r, azr, rzero, r * (1 - r)

Tests, hints, verified solutions, and AI help, with Pro at $7.99 a month.

Choosing a self-play scheme

The schemes differ in what converges, what you ship, and what each step costs.

scheme	converges in two-player zero-sum games	what you deploy	cost per step	where it is used
best response to the last self	no, it can cycle forever	the newest policy	one best response	naive baselines; fine in transitive games
AlphaZero-style self-play with search	empirically, in perfect-information games with a strong value function	the newest network plus search	many games, each with hundreds of simulations per move	Go, chess, shogi, AlphaProof
fictitious play	yes, the averages (Robinson 1951)	the average policy	one best response to the average	small games, and neural versions such as neural fictitious self-play (Heinrich and Silver 2016)
regret matching and counterfactual regret minimization	yes, the averages, within $O(1/\sqrt{T})$	the average policy	a regret update per decision point	poker
PSRO with a Nash meta-solver (double oracle)	yes, in finitely many steps in finite games	a mixture over the population	a reinforcement learning best response plus a growing payoff table	research on general games
leagues with exploiters	no guarantee, built to reduce exploitability	the main agent	many agents trained in parallel	AlphaStar

⚠️ Where self-play goes wrong

There are three classic implementation bugs. The backup forgets the sign flip. The outcome $z$ is recorded for one fixed player instead of for the player to move at each stored position. And illegal moves are left in the policy target.

There are three classic evaluation bugs. One is reporting the win rate against the previous checkpoint, which is 100 percent at every step of a cycle. Another is reporting the last strategy in a game where only the average converges. The third is treating Elo against humans as proof that a system cannot be exploited.

The classic design bug is self-play with a learned referee and no ground truth anywhere. It turns into an optimizer against the referee, as Language Self-Play's Python answers and R-Zero's falling label accuracy show.

🧭 What to remember

Self-play works when three things come together, an exact reward from the rules, an opponent that improves with you, and a search that makes the current policy stronger for one move.
MCTS with PUCT picks moves by average value plus a prior-weighted bonus for rarely tried moves. It judges new leaves with a value function and flips the sign at every level on the way back up. Its visit counts are a better policy than the prior, worth 2,130 Elo in Go.
AlphaZero trains one network on $(z - v)^2 - \boldsymbol\pi^{\text{search}\top}\log\mathbf{p} + c_{\text{reg}}\lVert\theta\rVert^2$ , with noise at the root so that every move gets explored. This is expert iteration, and at scale it is mostly inference.
In games where strength goes in circles, answering your last self cycles forever at maximal exploitability. Fictitious play and regret matching converge on average, because exploitability of the averages equals the sum of the two players' average regrets. Populations, PSRO, and leagues with exploiters are the versions for neural networks.
Superhuman is not unexploitable. An adversarial policy beat KataGo over 97 percent of the time, and every policy trained against a frozen learned judge is such an adversary.
For language, the opponent is easy and the reward is hard. SPIN's referee is its supervised data, so it converges to the data and no further. Debate made a judge more accurate. Proposer-solver loops reward tasks of middling difficulty and go furthest when a program checks the answers (Absolute Zero, self-play SWE-RL, AlphaProof), and they degrade when the model's own votes do (R-Zero).
Next, §26.3 takes expert iteration from games to reasoning, where sampling plus a checker of final answers plays the part of search, and derives exactly what filtered self-training optimizes.

Silver et al. (2017). Mastering the game of Go without human knowledge. AlphaGo Zero, with the full search algorithm, the training pipeline, and the raw-network-versus-search Elo comparison in the Methods.
Silver et al. (2017). Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm. AlphaZero, its loss (Equation 1), the Dirichlet noise settings, and the training statistics of Table S3. The 2018 Science version, A general reinforcement learning algorithm that masters chess, shogi, and Go through self-play, adds longer matches.
Anthony, Tian & Barber (2017). Thinking Fast and Slow with Deep Learning and Tree Search. Expert iteration on Hex, the template behind AlphaZero and STaR.
Lanctot et al. (2017). A Unified Game-Theoretic Approach to Multiagent Reinforcement Learning. PSRO, joint-policy correlation, and how iterated best response, fictitious play, and double oracle fit into one algorithm.
Wang et al. (2023). Adversarial Policies Beat Superhuman Go AIs. The KataGo attack, why it works, and why adversarial training did not remove it.
Chen et al. (2024). Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models. SPIN, Theorems 5.2 and 5.4, and the zephyr-7b experiments.
Zhao et al. (2025). Absolute Zero: Reinforced Self-play Reasoning with Zero Data. A proposer-solver loop grounded in a code executor, with the learnability reward of Equation 4.
Hubert et al. (2025). Olympiad-level formal mathematical reasoning with reinforcement learning. AlphaProof, the AlphaZero recipe with Lean as the rulebook.

Made for larger screens

Self-play

A game small enough to solve by hand

Tree search looks ahead only where it matters

Why the backup flips the sign

Four simulations by hand

Visit counts become a policy

Policy targets from visit counts

MCTS with PUCT for a two-player game

What search alone can and cannot see

Why imitating the search makes the network stronger

AlphaZero closes the loop

Four design choices and their reasons

What the loop costs at scale

The whole loop on Nim

When beating your last self goes in circles

Self-play that converges on average

Self-play that cycles and self-play that converges

Regret matching and the regret-to-Nash identity

Populations of past selves stand in for the average

Superhuman is not unexploitable

Language models have the opponent but not the referee

Games played in language

Self-play against your previous self

SPIN in a table

Debate, and games with a learned referee

Curricula that write their own problems

Learnability rewards for a proposer

Choosing a self-play scheme

Read more

Watch next