Speaker
Eric Jang
Appearances over time
1 episodes
Episodes
1
Podcasts
Quotes & moments
David Wu's open-source KataGo project in 2020 achieved a 40x reduction in the compute needed to train a strong Go bot tabula rasa.
A 19x19 Go board has roughly 361 possible moves per turn over ~300 moves, producing a game tree larger than the number of atoms in the universe.
Eric Jang trained a strong Go bot for roughly $10K of donated compute from Prime Intellect, spending about $3K on the final training run.
Unlike naive RL which must figure out which of 100K+ tokens caused a win, MCTS provides a strictly better action target for every single move in every game.
During AlphaGo training, MCTS uses between 200 and 2048 simulations per move; the AlphaGo Lee match used tens of thousands of simulations per move.
AlphaGo Lee (the original paper) used two separate networks for policy and value; all subsequent AlphaGo papers merged them into one network with two heads.
Eric Jang found that for small data regimes in Go, ResNets outperform Transformers, providing better bang for the buck at lower compute budgets.
AlphaGo Zero's training plot shows the first ~30 hours are spent just catching up to a supervised learning baseline before surpassing it.
Current LLM RL treats the entire token sequence as a single action (T=1), avoiding compounding variance but losing per-token credit assignment.
Distillation from soft targets provides far more bits per sample than one-hot labels because the entropy of a soft distribution is vastly higher.
Pre-training on 9x9 Go boards and then warm-starting a 19x19 model dramatically cuts the time needed to learn endgame value functions.
AlphaGo's elegance is that MCTS always produces a training signal — the policy is never stuck at zero pass rate waiting to stumble on a win.
Naive RL reinforces entire winning trajectories — but most of those moves were irrelevant. MCTS gives you a strictly better action label for every single move in every game. That's why AlphaGo learns far faster than an LLM-style reinforce loop.
A 19x19 Go board generates a game tree larger than atoms in the universe — 361 to the power of 300. This isn't just big, it's a fundamentally different class of problem from chess, which is why computer scientists thought Go was unsolvable this century.
MCTS doesn't build the whole tree — it builds it interactively while searching. Selection, expansion, evaluation, backup: these four steps run hundreds or thousands of times per move, concentrating compute on the branches that matter.
AlphaGo's neural network does two things: predict who's going to win (value head), and predict which moves are worth trying (policy head). Train this on human expert games and you already get a very strong Go player before any search.
MCTS doesn't just pick the best move — it produces a better probability distribution than the raw policy network. AlphaGo then trains the policy network to imitate that improved distribution. Iterate, and you get exponential improvement.
When you can't build a search tree — like in StarCraft — you fix your opponent and use model-free RL to find the best response. That best response becomes a better label for your current policy. Same idea as MCTS, different implementation.
Language has billions of possible next tokens — the PUCT exploration heuristic assumes you'll visit the same node multiple times, but an LLM will almost never generate the exact same token sequence twice. The discrete action assumption breaks.
With a 1-in-100K pass rate, supervised learning gives you negative log(1/100K) = ~17 bits per sample. Naive RL gives you essentially zero. You spend almost all of early training in a regime where nothing is learned.
A 10-layer neural network can compress what looks like an NP-class search problem into a single forward pass. This happened with Go, protein folding, and tensor decomposition. It might mean our understanding of computational hardness is fundamentally incomplete.
LLMs can autonomously run experiments, tune hyperparameters, and optimize code. What they can't do: decide which question to investigate next, or recognize when an entire research track is a dead end and pivot to something fundamentally different.
Off-policy data is fine — and useful — if those states are ones your current policy might visit. It becomes actively harmful when you're training on states your policy would never reach, wasting capacity on irrelevant corrections.
Scaling laws only emerge cleanly when your system already works, your data is good, and there are no bugs. Trying to extract scaling insights before you have a working baseline just gives you scaling laws on garbage.
Go has a fast, unambiguous outer loop (win rate vs. KataGo) and contains subtasks that overlap with LLMs and robotics — distributed systems, hyperparameter search, hypothesis generation. It's a cheap, verifiable testbed for automated AI research.
Analysis
What they talk about
- Technology 46%
- Science 36%
- Leisure 18%
Connections
Shows they appear on and people they share episodes with. Drag to explore.
