40x compute reduction (2020)
David Wu's open-source KataGo project in 2020 achieved a 40x reduction in the compute needed to train a strong Go bot tabula rasa.
Eric Jang – Building AlphaGo from scratch
A 10-layer neural network can compress a nearly intractable game-tree search — and AlphaGo's training loop is actually just supervised learning on better labels, never raw RL.
May 15, 2026
2:37:29
Difficulty: Intermediate
Played
Dwarkesh Podcast
Eric Jang – Building AlphaGo from scratch
A 10-layer neural network can compress a nearly intractable game-tree search — and AlphaGo's training loop is actually just supervised learning on better labels, never raw RL.
May 15, 2026
2:37:29
Difficulty: Intermediate
Played
TL;DR
Eric Jang, former VP of AI at 1X Technologies, walks through building AlphaGo from scratch using modern AI tools, using the project to illuminate the future of AI research. He explains how Monte Carlo Tree Search (MCTS) sidesteps the credit assignment problem that plagues naive policy gradient RL — by generating a strictly better action label for every single move [1] — Eric Jang "Naive RL reinforces entire winning trajectories — but most of those moves were irrelevant. MCTS gives you a strictly better action label fo…" 1:15:40 , rather than reinforcing entire winning trajectories. This contrast exposes deep inefficiencies in how LLMs currently learn via RL [2] — Dwarkesh Patel "With a 1-in-100K pass rate, supervised learning gives you negative log(1/100K) = ~17 bits per sample. Naive RL gives you essentially zero. …" 2:09:35 . The key takeaway: AlphaGo's self-play loop is elegant precisely because it never starts from zero signal — MCTS always gives you something to improve on [3] — Eric Jang "A 10-layer neural network can compress what looks like an NP-class search problem into a single forward pass. This happened with Go, protei…" 1:51:30 .
#AlphaGo
#Monte Carlo Tree Search
#reinforcement learning
#LLM RL training
#credit assignment problem
#policy gradient variance
#self-play
#AI research automation
#scaling laws
#NP-hard problems and neural nets
#distillation
#off-policy learning
#Go AI
#inference-time compute
#bits per sample
#MCTS
#LLMs
#credit assignment
#policy gradient
#value function
#neural networks
#Go
#automated research
#KataGo
#off-policy training
#ResNets
#exploration
#AlphaZero
#NP-hard
Eric Jang walks through how to build AlphaGo from scratch using modern AI tools, explaining MCTS, self-play, and RL — and drawing connections to how LLMs learn and where AI research automation is headed.
Chapter list
-
Dwarkesh explains how he used the Cursor agent SDK to build a multi-stage flashcard generation pipeline for this episode, including blackboard screenshot ingestion and SVG generation.
-
Eric walks through the UCB1 bandit algorithm and AlphaGo's PUCT variant — how visit counts, Q-values, and prior probabilities are balanced to explore and exploit.
-
Eric describes the neural network architecture — ResNet or Transformer, two output heads (policy and value) — and explains how training on expert human games produces a strong Go player before any search. [1] — Eric Jang "AlphaGo's neural network does two things: predict who's going to win (value head), and predict which moves are worth trying (policy head). …" 32:00
-
Eric explains how MCTS produces a more confident action distribution than the raw policy, and how AlphaGo trains the policy to imitate that improved distribution — the core self-play loop. [1] — Eric Jang "MCTS doesn't just pick the best move — it produces a better probability distribution than the raw policy network. AlphaGo then trains the p…" 1:01:00
-
Eric explains neural fictitious self-play — training best-response policies against fixed opponents and distilling them — as an MCTS substitute for games without tractable tree search.
-
Eric and Dwarkesh discuss why MCTS fails for LLM reasoning: language's vast token space violates the discrete finite-action assumption, and value estimation is much harder than in Go. [1] — Eric Jang "Language has billions of possible next tokens — the PUCT exploration heuristic assumes you'll visit the same node multiple times, but an LL…" 1:44:55
-
Eric explains the DAGGER-inspired view of replay buffers: off-policy data is helpful if it covers reachable states, harmful if it covers states the current policy would never visit. [1] — Eric Jang "Off-policy data is fine — and useful — if those states are ones your current policy might visit. It becomes actively harmful when you're tr…" 2:01:20
-
Dwarkesh presents an information-theoretic argument: naive RL learns near-zero bits per sample at low pass rates, while supervised learning learns negative log(pass rate) bits. The gap is enormous. [1] — Dwarkesh Patel "With a 1-in-100K pass rate, supervised learning gives you negative log(1/100K) = ~17 bits per sample. Naive RL gives you essentially zero. …" 2:09:35
-
Eric reports from his AlphaGo project: LLMs excel at hyperparameter tuning and executing experiments, but fail at lateral thinking — choosing the right question and recognizing dead-end research tracks. [1] — Eric Jang "LLMs can autonomously run experiments, tune hyperparameters, and optimize code. What they can't do: decide which question to investigate ne…" 2:14:00
- MCTS (Monte Carlo Tree Search)
- A tree-search algorithm that iteratively selects, expands, evaluates, and backs up nodes to estimate the best move, without needing to search the full game tree.
- PUCT
- Predicted Upper Confidence Bound with Trees — the action selection formula in AlphaGo that balances exploitation (mean action value Q) with exploration (prior probability P divided by visit count).
- Policy network
- A neural network that outputs a probability distribution over possible moves, representing which actions are likely to be good from a given board state.
- Value network
- A neural network that takes a board state and predicts the probability of winning from that state, allowing deep tree search to be truncated early.
- Credit assignment problem
- The challenge in RL of determining which specific actions in a long trajectory were responsible for a final reward, as opposed to irrelevant or harmful actions.
- Policy gradient (REINFORCE)
- A family of RL algorithms that estimate gradients by rolling out full trajectories and scaling action log-probabilities by the trajectory's return, suffering from high variance.
- Tabula rasa
- Latin for 'blank slate' — in AI, training a model entirely from self-play or random initialization without any supervised pretraining on human data.
- Tromp-Taylor rules
- A version of Go rules designed to be completely unambiguous and algorithmically decidable, used by all Go AIs for scoring instead of the human-interpretation-dependent Japanese or Chinese rules.
- DAGGER
- Dataset Aggregation — an imitation learning algorithm that corrects a policy by querying an expert for better actions at states the current policy visits, used here as an analogy to MCTS relabeling.
- Off-policy training
- Training a reinforcement learning agent on data collected by a different (older) version of the policy, rather than the current policy — can be unstable if the old data is too far from the current policy's distribution.
- Advantage estimation
- In RL, the advantage of an action is how much better it is than the average — subtracting a baseline value from the return to reduce gradient variance.
- Neural fictitious self-play
- A game-theoretic training method where agents are trained as best responses to a league of fixed opponents, then distilled into a mixed strategy — used in AlphaStar and OpenAI Five.
- TD learning (Temporal Difference)
- A class of RL methods that update value estimates using the difference between predicted and observed rewards at successive timesteps, without waiting for full episode outcomes.
- Amortize
- To spread the cost of a computation across many instances; here, a neural network 'amortizes' expensive tree search by encoding its results into learned weights.
- Inductive bias
- The set of assumptions a model makes about data structure; CNNs have a local spatial bias, while Transformers have a global attention bias — relevant to choosing architectures for Go.
- Lorenz attractor
- A canonical chaotic dynamical system whose trajectories are unpredictable in detail but remain on a recognizable butterfly-shaped structure — used here as an analogy for macroscopic predictability despite microscopic chaos.
- Bitter lesson
- Richard Sutton's observation that in the long run, general methods that leverage compute (search, learning) consistently outperform methods that encode human knowledge — referenced multiple times as a design principle.
- KL minimization
- Kullback-Leibler divergence minimization — a training objective that measures how much one probability distribution differs from another; used in AlphaGo to train the policy to match the MCTS distribution.
- Bellman backup
- The dynamic programming operation that updates a value or Q-function estimate using the reward plus the discounted value of the next state — the backbone of Q-learning and TD methods.
- Replay buffer
- A data structure that stores past transitions (state, action, reward, next state) for off-policy RL training, allowing the agent to learn from experiences not generated by its current policy.
Chapter 1 · 00:00
Basics of Go
Dwarkesh explains how he used the Cursor agent SDK to build a multi-stage flashcard generation pipeline for this episode, including blackboard screenshot ingestion and SVG generation.
Chapter 2 · 08:17
Monte Carlo Tree Search
Eric walks through the UCB1 bandit algorithm and AlphaGo's PUCT variant — how visit counts, Q-values, and prior probabilities are balanced to explore and exploit.
Why Go Is a Hard Problem for Computers
A 19x19 Go board generates a game tree larger than atoms in the universe — 361 to the power of 300. This isn't just big, it's a fundamentally different class of problem from chess, which is why computer scientists thought Go was unsolvable this century.
361^300 possible game trees
A 19x19 Go board has roughly 361 possible moves per turn over ~300 moves, producing a game tree larger than the number of atoms in the universe.
MCTS: How AlphaGo Thinks Before Moving
MCTS doesn't build the whole tree — it builds it interactively while searching. Selection, expansion, evaluation, backup: these four steps run hundreds or thousands of times per move, concentrating compute on the branches that matter.
The Policy and Value Networks: What They Do
AlphaGo's neural network does two things: predict who's going to win (value head), and predict which moves are worth trying (policy head). Train this on human expert games and you already get a very strong Go player before any search.
Chapter 3 · 32:04
What the neural network does
Eric describes the neural network architecture — ResNet or Transformer, two output heads (policy and value) — and explains how training on expert human games produces a strong Go player before any search. [1] — Eric Jang "AlphaGo's neural network does two things: predict who's going to win (value head), and predict which moves are worth trying (policy head). …" 32:00
ResNets outperform Transformers at low budget
Eric Jang found that for small data regimes in Go, ResNets outperform Transformers, providing better bang for the buck at lower compute budgets.
200–2048 simulations per move
During AlphaGo training, MCTS uses between 200 and 2048 simulations per move; the AlphaGo Lee match used tens of thousands of simulations per move.
Chapter 4 · 1:00:33
Self-play
Eric explains how MCTS produces a more confident action distribution than the raw policy, and how AlphaGo trains the policy to imitate that improved distribution — the core self-play loop. [1] — Eric Jang "MCTS doesn't just pick the best move — it produces a better probability distribution than the raw policy network. AlphaGo then trains the p…" 1:01:00
MCTS as a Policy Improvement Operator
MCTS doesn't just pick the best move — it produces a better probability distribution than the raw policy network. AlphaGo then trains the policy network to imitate that improved distribution. Iterate, and you get exponential improvement.
Why MCTS Beats Naive RL for Credit Assignment
Naive RL reinforces entire winning trajectories — but most of those moves were irrelevant. MCTS gives you a strictly better action label for every single move in every game. That's why AlphaGo learns far faster than an LLM-style reinforce loop.
Chapter 5 · 1:25:38
Alternative RL approaches
Eric explains neural fictitious self-play — training best-response policies against fixed opponents and distilling them — as an MCTS substitute for games without tractable tree search.
Neural Fictitious Self-Play: How AlphaStar Learned Without Trees
When you can't build a search tree — like in StarCraft — you fix your opponent and use model-free RL to find the best response. That best response becomes a better label for your current policy. Same idea as MCTS, different implementation.
AlphaGo Lee: 2 networks merged
AlphaGo Lee (the original paper) used two separate networks for policy and value; all subsequent AlphaGo papers merged them into one network with two heads.
LLMs treat full trajectory as 1 action
Current LLM RL treats the entire token sequence as a single action (T=1), avoiding compounding variance but losing per-token credit assignment.
Chapter 6 · 1:41:09
Why doesn't MCTS work for LLMs
Eric and Dwarkesh discuss why MCTS fails for LLM reasoning: language's vast token space violates the discrete finite-action assumption, and value estimation is much harder than in Go. [1] — Eric Jang "Language has billions of possible next tokens — the PUCT exploration heuristic assumes you'll visit the same node multiple times, but an LL…" 1:44:55
Scaling Laws Require a Working System First
Scaling laws only emerge cleanly when your system already works, your data is good, and there are no bugs. Trying to extract scaling insights before you have a working baseline just gives you scaling laws on garbage.
MCTS: one better label per move
Unlike naive RL which must figure out which of 100K+ tokens caused a win, MCTS provides a strictly better action target for every single move in every game.
Why MCTS Doesn't Work for LLMs
Language has billions of possible next tokens — the PUCT exploration heuristic assumes you'll visit the same node multiple times, but an LLM will almost never generate the exact same token sequence twice. The discrete action assumption breaks.
A 10-Layer Network Solves What Seemed NP-Hard
A 10-layer neural network can compress what looks like an NP-class search problem into a single forward pass. This happened with Go, protein folding, and tensor decomposition. It might mean our understanding of computational hardness is fundamentally incomplete.
$10K total compute spend
Eric Jang trained a strong Go bot for roughly $10K of donated compute from Prime Intellect, spending about $3K on the final training run.
AlphaGo Zero: ~3e23 flops
AlphaGo Zero was trained on far more compute than any other AI model of its era — roughly 3e23 flops, comparable in order of magnitude to frontier LLMs.
9x9→19x19 transfer learning
Pre-training on 9x9 Go boards and then warm-starting a 19x19 model dramatically cuts the time needed to learn endgame value functions.
AlphaGo Zero: first 30 hrs catching up to SL
AlphaGo Zero's training plot shows the first ~30 hours are spent just catching up to a supervised learning baseline before surpassing it.
Chapter 7 · 2:01:09
Off-policy training
Eric explains the DAGGER-inspired view of replay buffers: off-policy data is helpful if it covers reachable states, harmful if it covers states the current policy would never visit. [1] — Eric Jang "Off-policy data is fine — and useful — if those states are ones your current policy might visit. It becomes actively harmful when you're tr…" 2:01:20
Off-Policy Training: When It Helps and When It Hurts
Off-policy data is fine — and useful — if those states are ones your current policy might visit. It becomes actively harmful when you're training on states your policy would never reach, wasting capacity on irrelevant corrections.
Chapter 8 · 2:08:56
RL is even more information inefficient than you thought
Dwarkesh presents an information-theoretic argument: naive RL learns near-zero bits per sample at low pass rates, while supervised learning learns negative log(pass rate) bits. The gap is enormous. [1] — Dwarkesh Patel "With a 1-in-100K pass rate, supervised learning gives you negative log(1/100K) = ~17 bits per sample. Naive RL gives you essentially zero. …" 2:09:35
RL Is Even More Inefficient Than You Think
With a 1-in-100K pass rate, supervised learning gives you negative log(1/100K) = ~17 bits per sample. Naive RL gives you essentially zero. You spend almost all of early training in a regime where nothing is learned.
RL bits per sample vs. SL
Naive RL learns at the rate of the entropy of a binary random variable per sample, while supervised learning learns negative log(pass rate) bits — orders of magnitude more at low pass rates.
What Automated AI Research Can and Can't Do Yet
LLMs can autonomously run experiments, tune hyperparameters, and optimize code. What they can't do: decide which question to investigate next, or recognize when an entire research track is a dead end and pivot to something fundamentally different.
MCTS always in non-zero signal regime
AlphaGo's elegance is that MCTS always produces a training signal — the policy is never stuck at zero pass rate waiting to stumble on a win.
Chapter 9 · 2:16:16
Automated AI researchers
Eric reports from his AlphaGo project: LLMs excel at hyperparameter tuning and executing experiments, but fail at lateral thinking — choosing the right question and recognizing dead-end research tracks. [1] — Eric Jang "LLMs can autonomously run experiments, tune hyperparameters, and optimize code. What they can't do: decide which question to investigate ne…" 2:14:00
Soft label distillation entropy advantage
Distillation from soft targets provides far more bits per sample than one-hot labels because the entropy of a soft distribution is vastly higher.
Go as a Research Environment for AI
Go has a fast, unambiguous outer loop (win rate vs. KataGo) and contains subtasks that overlap with LLMs and robotics — distributed systems, hyperparameter search, hypothesis generation. It's a cheap, verifiable testbed for automated AI research.
No indexed bits in this chapter.
Show stoppers
Why MCTS Beats Naive RL for Credit Assignment
Naive RL reinforces entire winning trajectories — but most of those moves were irrelevant. MCTS gives you a strictly better action label for every single move in every game. That's why AlphaGo learns far faster than an LLM-style reinforce loop.
A 10-Layer Network Solves What Seemed NP-Hard
A 10-layer neural network can compress what looks like an NP-class search problem into a single forward pass. This happened with Go, protein folding, and tensor decomposition. It might mean our understanding of computational hardness is fundamentally incomplete.
RL Is Even More Inefficient Than You Think
With a 1-in-100K pass rate, supervised learning gives you negative log(1/100K) = ~17 bits per sample. Naive RL gives you essentially zero. You spend almost all of early training in a regime where nothing is learned.
Snapshots ()
Key Quotes ()
This episode
Claims & Sources
6 / 14 cited (43%)
Factual claims made this episode, and whether a source was named.
David Wu's KataGo project achieved a 40x reduction in compute needed to train a strong tabula rasa Go bot compared to previous work.
A 19x19 Go board has approximately 361 possible moves per turn over roughly 250-300 moves per game, producing a tree larger than the number of atoms in the universe.
AlphaGo Lee (the original AlphaGo paper) used two separate neural networks for policy and value; all subsequent AlphaGo papers merged them into a single network with two output heads.
AlphaGo Zero was trained on approximately 3e23 flops — an anomalous outlier in the historical chart of AI training compute, comparable in order of magnitude to frontier LLMs.
Eric Jang trained a competitive AlphaGo-style Go bot for approximately $10K in donated compute from Prime Intellect, with ~$3K spent on the final training run.
In AlphaGo Lee, the value estimate used during MCTS was averaged with the result of a full playout to the end of the game; all subsequent AlphaGo papers removed this and used only the neural network value.
For low-budget experiments on Go, ResNets outperform Transformers because of their local convolutional inductive bias, though this may reverse at higher data scales.
KataGo found it useful to pool global features throughout the network to give it a global sense of how to connect value across different parts of the 19x19 board.
Andy Jones' 2021 paper 'Scaling Scaling Laws for Board Games' showed that training compute and inference compute (MCTS simulations) can be traded off, anticipating inference-time scaling laws for LLMs.
AlphaGo Zero's training plot shows that approximately the first 30 hours are spent just catching up to the supervised learning baseline before the tabula rasa policy surpasses it.
In naive policy gradient RL, gradient variance grows quadratically with trajectory length due to the multiplicative coupling between reward and per-token log-probability terms.
Current LLM RL treats the entire decoded token sequence as a single action (T=1), avoiding multi-step gradient variance but losing fine-grained credit assignment.
In perfect information games like Go, there exists a Nash equilibrium strategy decidable solely from the current state — no temporal history of opponent moves is required.
Distillation from soft target distributions is far more information-efficient per sample than one-hot labels because soft distributions have much higher entropy.
This episode
Cast
-
Referenced for his description of policy gradient RL as 'sucking supervision through a straw', and as a podcast guest.
-
Author of the 2021 paper 'Scaling Scaling Laws for Board Games', which anticipated inference-compute scaling tradeoffs.
-
Jane Street employee and author of the open-source KataGo Go bot, which achieved a 40x compute reduction over AlphaGo.
-
The AI lab that originally developed AlphaGo; Eric Jang previously worked at Google DeepMind Robotics.
-
Quantitative trading firm where KataGo author David Wu works; also a podcast sponsor that gave Dwarkesh a datacenter tour.
-
Robotics company where Eric Jang was most recently VP of AI before his sabbatical.
-
Provided Eric Jang with approximately $10K in donated compute for his AlphaGo project.
-
DeepMind's Go-playing AI system, the central topic of the episode — its architecture, training, and lessons for future AI.
-
Open-source Go bot by David Wu that achieved a 40x compute reduction and is used as the benchmark opponent in Eric Jang's project.
-
Successor to AlphaGo Lee that learned tabula rasa without human game data, and was trained on anomalously high compute (~3e23 flops).
-
Episode sponsor; Dwarkesh used Cursor's agent SDK to build a flashcard generation pipeline for the episode.
-
DeepMind's protein structure prediction system, cited as another example of a neural network compressing an NP-class simulation problem.
-
DeepMind's StarCraft-playing AI, discussed as an example of neural fictitious self-play applied to a game without tractable tree search.
-
DeepMind's matrix multiplication AI, cited as another instance of neural networks solving apparently intractable optimization problems.
-
OpenAI's Dota-playing AI, mentioned alongside AlphaStar as a system that used neural fictitious self-play.
Stats
