Where this was said
RL is even more information inefficient than you thought
At 2:14:38 · chapter starts 2:08:56
Dwarkesh presents an information-theoretic argument: naive RL learns near-zero bits per sample at low pass rates, while supervised learning learns negative log(pass rate) bits. The gap is enormous. [1] — Dwarkesh Patel "With a 1-in-100K pass rate, supervised learning gives you negative log(1/100K) = ~17 bits per sample. Naive RL gives you essentially zero. …" 2:09:35
With a 1-in-100K pass rate, supervised learning gives you negative log(1/100K) = ~17 bits per sample. Naive RL gives you essentially zero. You spend almost all of early training in a regime where nothing is learned.
Naive RL learns at the rate of the entropy of a binary random variable per sample, while supervised learning learns negative log(pass rate) bits — orders of magnitude more at low pass rates.
LLMs can autonomously run experiments, tune hyperparameters, and optimize code. What they can't do: decide which question to investigate next, or recognize when an entire research track is a dead end and pivot to something fundamentally different.
AlphaGo's elegance is that MCTS always produces a training signal — the policy is never stuck at zero pass rate waiting to stumble on a win.
