Quote · Dwarkesh Podcast
The next big breakthrough will be AIs learning on the job
Where this was said
Getting the learning back to the weights
At 9:03 · chapter starts 8:41
Dwarkesh explores on-policy self-distillation (OPSD) as the leading technique for continual learning, contrasting it with naive SFT and explaining why RL's sparse parameter updates are a feature, not a bug [1] — Dwarkesh Patel "OPSD trains the base model to match what a veteran 'teacher' model — loaded with a full session's context — would predict. It needs no oute…" 11:05 .
The CursorTab model online-learns by predicting which edits users accept, processing over 400 million requests per day.
OPSD trains the base model to match what a veteran 'teacher' model — loaded with a full session's context — would predict. It needs no outer-loop reward, provides denser per-token supervision than RL, and avoids overwriting existing knowledge. It's the most credible path to continual learning today.
Unlike RL-VR, on-policy self-distillation doesn't require an outer-loop verifiable reward — it only requires a model that can learn the right things within the context window.
On-policy self-distillation provides per-token supervision rather than a single end-of-trajectory reward, making it a much denser training signal than naive RL.
Naively fine-tuning on session transcripts trains models to recall everything with perfect fidelity — but that's not how skill acquisition works. RL concentrates updates only where they matter and changes very few parameters, which is exactly what you want when you don't want to forget what you already know.
RL training changes very few model parameters per step, concentrating updates only on what is relevant to achieving the outcome — a crucial property for continual learning.
Dreaming means the model spends compute generating its own simulated RL environments, then trains against them — rehearsing skills relevant to the actual user's real-world context. Like EfficientZero playing dozens of internal games per real step, this could make AI far more sample-efficient without requiring external data.
