AI models are approximately 1/1,000,000th as sample efficient as humans during training.
The next big breakthrough will be AIs learning on the job
By 2027, AI models may improve faster from real-world deployment experience than from any pre-training run — and that shift will be both scary and transformative.
Jun 26, 2026
19:53
Difficulty: Intermediate
Played
Dwarkesh Podcast
The next big breakthrough will be AIs learning on the job
By 2027, AI models may improve faster from real-world deployment experience than from any pre-training run — and that shift will be both scary and transformative.
Jun 26, 2026
19:53
Difficulty: Intermediate
Played
TL;DR
Dwarkesh Patel argues that the next frontier in AI isn't just scaling RLVR training on verifiable tasks, but enabling AIs to learn continuously on the job from real-world deployment [1] — Dwarkesh Patel "The labs' core AGI bet is that training on enough verifiable, containerized RL environments will produce agents capable of open-ended probl…" . He identifies "grindability" — the ability to run parallel rollouts in deterministic simulators — as an underrated bottleneck, explaining why computer use lags behind coding [2] — Dwarkesh Patel "Many skills humans have — winning court cases, building businesses, day trading — require real-world interaction with sparse, delayed feedb…" 05:30 . Two speculative solutions emerge: on-policy self-distillation (OPSD) to compress session learnings back into model weights [3] — Dwarkesh Patel "OPSD trains the base model to match what a veteran 'teacher' model — loaded with a full session's context — would predict. It needs no oute…" 11:05 , and "dreaming," where AIs build their own simulated environments to rehearse skills [4] — Dwarkesh Patel "Dreaming means the model spends compute generating its own simulated RL environments, then trains against them — rehearsing skills relevant…" 14:00 . The key takeaway: by 2027–28, AI improvement may come primarily from deployment experience, not pre-training.
#continual learning
#RLVR generalization
#on-policy self-distillation
#AI sample efficiency
#grindability
#computer use AI
#test-time training
#AI scaling
#on-the-job learning
#EfficientZero
#inference compute waste
#AI 2027 forecast
#RLVR
#dreaming
#sample efficiency
#AGI
#computer use
#inference compute
#context window
#online learning
#RL generalization
#AI deployment
Dwarkesh Patel argues that the next frontier in AI is enabling models to learn continuously from real-world deployment — and explores grindability, on-policy self-distillation, and 'dreaming' as the keys to getting there.
Chapter list
-
Dwarkesh outlines the core labs' bet: scale verifiable RL training across millions of tasks to produce a general problem-solving agent. He also raises the counterargument that AI is 1/1,000,000th as sample efficient as humans [1] — Dwarkesh Patel "1/1,000,000th sample efficiency: Current AI models are one-millionth as sample efficient as humans during training, a deficit Dwarkesh prev…" 00:42 .
-
Dwarkesh introduces 'grindability' — the ability to run parallel rollouts in deterministic simulators — as the underrated reason computer use AI lags far behind coding, despite being equally verifiable [1] — Dwarkesh Patel "Verifiability alone doesn't explain which domains see rapid AI progress. The real bottleneck is grindability: can you run thousands of para…" 02:12 .
-
Dwarkesh questions whether RLVR generalizes from containerized tasks to complex real-world domains like politics or business, citing Dario Amodei's context-length degradation comment as a key data point [1] — Dwarkesh Patel "Dario Amodei noted that training at short context lengths and serving at long ones can cause performance degradation. Dwarkesh reads this a…" 06:55 .
-
Dwarkesh explores on-policy self-distillation (OPSD) as the leading technique for continual learning, contrasting it with naive SFT and explaining why RL's sparse parameter updates are a feature, not a bug [1] — Dwarkesh Patel "OPSD trains the base model to match what a veteran 'teacher' model — loaded with a full session's context — would predict. It needs no oute…" 11:05 .
-
Dwarkesh introduces 'dreaming' — AI building its own RL environments to rehearse skills — using EfficientZero as an analogy, and positions it as a potential fourth scaling axis alongside pre-training, RL, and inference compute [1] — Dwarkesh Patel "Dreaming means the model spends compute generating its own simulated RL environments, then trains against them — rehearsing skills relevant…" 14:00 .
-
Dwarkesh sketches a concrete 2027–28 scenario where RLVR-primed agents get week-long context windows, do real work, and distill session learnings back into weights — enabling AI to improve primarily from deployment experience [1] — Dwarkesh Patel "RLVR produces an agent competent enough to deploy. That agent gets a week-long context window and does real work. At the end of the week, a…" 16:30 .
- RLVR
- Reinforcement Learning from Verifiable Rewards — a training paradigm where AI agents receive reward signals only from tasks with objectively checkable correct answers, like coding or math.
- Grindability
- Dwarkesh Patel's coined term for a domain's suitability for massive parallel RL training — requiring deterministic, replayable simulations that can be run from identical starting points thousands of times simultaneously.
- On-policy self-distillation (OPSD)
- A training technique where a base model is trained to match the predictions of a 'teacher' version of itself that has accumulated a full session's context, distilling session learnings back into the base weights.
- Dreaming
- A speculative AI capability where a model generates its own simulated RL training environments and trains against them, rehearsing real-world skills without requiring external data.
- Continual learning
- The ability of a model to update its weights — and thus permanently improve — from experience accumulated during deployment, rather than only from a discrete pre-training or fine-tuning run.
- KV-cache
- Key-Value cache — a memory structure in transformer models that stores intermediate attention computations to speed up inference, sometimes used as a proxy for short-term in-context memory.
- In-context learning
- A model's ability to adapt its behavior based on examples or information provided within a single input context window, without updating its underlying weights.
- Test-time training
- Updating a model's weights at inference time, using information encountered during a specific deployment session, rather than only during a fixed pre-training phase.
- Non-stationary environment
- In RL, an environment whose dynamics or reward structure change over time, making it harder to learn from fixed replay because past experience may not reflect current conditions.
- EfficientZero
- A model trained to be highly data-efficient in game-playing by internally simulating many hypothetical game steps for each real interaction, achieving human-competitive performance with far fewer real samples.
- Sparse attention
- A transformer architecture modification that attends to only a subset of tokens rather than all tokens in the context, improving memory efficiency for very long sequences.
- Grokking
- A phenomenon in ML where a model suddenly achieves strong generalization after extended training, long after it has memorized the training data — associated with deep representational compression.
- Rollout
- In RL, a single trajectory of an agent acting in an environment from a starting state through a sequence of actions and observations to a terminal state or reward.
- Amortized
- Spreading a fixed cost across many instances of use; here used to argue that the high one-time compute cost of pre-training is justified because it is reused across billions of inference sessions.
- Tacit knowledge
- Knowledge that is difficult to articulate or transfer explicitly — practical know-how embedded in experience, such as organizational norms or craft intuitions — as opposed to codified, explicit knowledge.
Chapter 1 · 00:00
The big research bet the labs are making
Dwarkesh outlines the core labs' bet: scale verifiable RL training across millions of tasks to produce a general problem-solving agent. He also raises the counterargument that AI is 1/1,000,000th as sample efficient as humans [1] — Dwarkesh Patel "1/1,000,000th sample efficiency: Current AI models are one-millionth as sample efficient as humans during training, a deficit Dwarkesh prev…" 00:42 .
Claims made here
People often report that employees are not net productive until 6 months or more into a job, indicating that on-the-job learning is necessary for competence.
The Big Research Bet: RLVR Will Build AGI
The labs' core AGI bet is that training on enough verifiable, containerized RL environments will produce agents capable of open-ended problem-solving for weeks on end. Skeptics argue the fundamental deficits — data inefficiency, no continual learning — will simply be steamrolled by scale, just as compute steamrolled NLP.
1/1,000,000th sample efficiency
Current AI models are one-millionth as sample efficient as humans during training, a deficit Dwarkesh previously explored in depth.
6 months on-the-job learning
People often say employees aren't net productive until 6+ months on the job, illustrating why continual learning matters for competence.
Chapter 2 · 02:12
Grindability is just as important as verifiability
Dwarkesh introduces 'grindability' — the ability to run parallel rollouts in deterministic simulators — as the underrated reason computer use AI lags far behind coding, despite being equally verifiable [1] — Dwarkesh Patel "Verifiability alone doesn't explain which domains see rapid AI progress. The real bottleneck is grindability: can you run thousands of para…" 02:12 .
Grindability: The Underrated Bottleneck in AI Progress
Verifiability alone doesn't explain which domains see rapid AI progress. The real bottleneck is grindability: can you run thousands of parallel rollouts from identical starting points in a deterministic simulator? Coding can; live websites cannot.
Computer use progress much slower than coding
Computer use has made far slower progress than coding and math despite being clearly verifiable, because it lacks grindable, deterministic training environments.
Grindability as a key bottleneck
It's not enough for a domain to be verifiable — it must also be 'grindable', meaning you can run thousands of parallel rollouts from identical starting points in a deterministic simulator.
Domains AI Can't Easily Learn: Politics, Trading, Court Cases
Many skills humans have — winning court cases, building businesses, day trading — require real-world interaction with sparse, delayed feedback. There's no containerized RL environment for 'build a company as well as Sam Walton.' These domains will resist the current training paradigm.
Chapter 3 · 06:10
Will RLVR alone generalize?
Dwarkesh questions whether RLVR generalizes from containerized tasks to complex real-world domains like politics or business, citing Dario Amodei's context-length degradation comment as a key data point [1] — Dwarkesh Patel "Dario Amodei noted that training at short context lengths and serving at long ones can cause performance degradation. Dwarkesh reads this a…" 06:55 .
Claims made here
Training AI at a short context length and then serving at a longer context length can lead to performance degradation.
Short-horizon RL training does not necessarily generalize to long-horizon RL performance, as suggested by context-length degradation in deployed models.
Around 30 to 50% of an AI lab's compute budget goes to inference, currently without producing any model improvement.
Dario Amodei's Hint That RLVR Generalization Has Limits
Dario Amodei noted that training at short context lengths and serving at long ones can cause performance degradation. Dwarkesh reads this as evidence that RLVR generalization is not infinite — a crack in the foundation of the labs' AGI bet.
30–50% of Compute Is Wasted on Inference
Labs burn 30–50% of compute on inference — and currently gain zero improvement to the model from it. Meanwhile, deployment is where the most valuable learning signals actually exist: real organizational context, real mistakes, real feedback.
Data point
50%
Labs spend 30 to 50% of their total compute on inference, yet this compute currently plays no productive role in improving the model.
Chapter 4 · 08:41
Getting the learning back to the weights
Dwarkesh explores on-policy self-distillation (OPSD) as the leading technique for continual learning, contrasting it with naive SFT and explaining why RL's sparse parameter updates are a feature, not a bug [1] — Dwarkesh Patel "OPSD trains the base model to match what a veteran 'teacher' model — loaded with a full session's context — would predict. It needs no oute…" 11:05 .
Claims made here
Some autistic savant individuals can recall random tables of numbers or nonsense syllables years later with extremely high fidelity, but this volume of recall impairs their ability to understand abstractions and metaphors.
The CursorTab model online-learns by predicting which code edits get accepted by users, processing over 400 million requests per day.
OPSD provides a denser supervision signal than naive RL by training on per-token probability discrepancies between a teacher and student model, rather than projecting a single reward through the whole trajectory.
Very few model parameters are actually changed during a single RL training step, concentrating the update only on what is relevant to achieving the outcome.
In a previous post, Dwarkesh argued that RL learns much less information per sample than supervised learning.
EfficientZero, a data-efficient game-playing model developed after AlphaZero, could beat a novice human at an unseen Atari game given only 2 hours of play, by internally simulating dozens of games per real step.
400M+ CursorTab requests/day
The CursorTab model online-learns by predicting which edits users accept, processing over 400 million requests per day.
On-Policy Self-Distillation (OPSD): Compressing Sessions Into Weights
OPSD trains the base model to match what a veteran 'teacher' model — loaded with a full session's context — would predict. It needs no outer-loop reward, provides denser per-token supervision than RL, and avoids overwriting existing knowledge. It's the most credible path to continual learning today.
OPSD requires no outer-loop reward
Unlike RL-VR, on-policy self-distillation doesn't require an outer-loop verifiable reward — it only requires a model that can learn the right things within the context window.
OPSD: denser supervision signal
On-policy self-distillation provides per-token supervision rather than a single end-of-trajectory reward, making it a much denser training signal than naive RL.
Why SFT Is a Bad Fit for On-the-Job Learning
Naively fine-tuning on session transcripts trains models to recall everything with perfect fidelity — but that's not how skill acquisition works. RL concentrates updates only where they matter and changes very few parameters, which is exactly what you want when you don't want to forget what you already know.
Very few parameters changed per RL step
RL training changes very few model parameters per step, concentrating updates only on what is relevant to achieving the outcome — a crucial property for continual learning.
Dreaming: AI Builds Its Own Training Simulations
Dreaming means the model spends compute generating its own simulated RL environments, then trains against them — rehearsing skills relevant to the actual user's real-world context. Like EfficientZero playing dozens of internal games per real step, this could make AI far more sample-efficient without requiring external data.
EfficientZero beats novice humans in 2h
EfficientZero, trained to be data-efficient, could beat a novice human at an unseen Atari game given only 2 hours of play time.
Chapter 5 · 15:23
Dreaming
Dwarkesh introduces 'dreaming' — AI building its own RL environments to rehearse skills — using EfficientZero as an analogy, and positions it as a potential fourth scaling axis alongside pre-training, RL, and inference compute [1] — Dwarkesh Patel "Dreaming means the model spends compute generating its own simulated RL environments, then trains against them — rehearsing skills relevant…" 14:00 .
Claims made here
Dreaming — where AI models generate and train against self-created simulated environments — could become a fourth axis of scaling alongside pre-training, RL, and inference-time compute.
4th axis of scaling: dreaming
Dreaming — where a model generates and trains against its own simulated environments — could become a fourth scaling axis alongside pre-training, RL, and inference-time compute.
What 2027 Looks Like: AI Learning on the Job
RLVR produces an agent competent enough to deploy. That agent gets a week-long context window and does real work. At the end of the week, a thumbs up triggers OPSD or dreaming to distill everything it learned back into the base weights. Repeat. The AI's skills start expanding beyond its original training domains — with each cycle.
Full-week co-work context window by 2027
Dwarkesh envisions that by 2027–28, effective context lengths will expand enough for AIs to co-work with a human for a full week of wall-clock time.
Chapter 6 · 17:23
What 2027 looks like
Dwarkesh sketches a concrete 2027–28 scenario where RLVR-primed agents get week-long context windows, do real work, and distill session learnings back into weights — enabling AI to improve primarily from deployment experience [1] — Dwarkesh Patel "RLVR produces an agent competent enough to deploy. That agent gets a week-long context window and does real work. At the end of the week, a…" 16:30 .
The Network Effect of AI Learning: Every User Makes It Smarter
Once continual learning arrives, AI improvement will stop being driven by pre-training runs. Instead, every user interaction globally feeds back into the model. It's a network effect applied to intelligence — and it's fundamentally different from anything that exists today.
No indexed bits in this chapter.
Show stoppers
Grindability: The Underrated Bottleneck in AI Progress
Verifiability alone doesn't explain which domains see rapid AI progress. The real bottleneck is grindability: can you run thousands of parallel rollouts from identical starting points in a deterministic simulator? Coding can; live websites cannot.
The Network Effect of AI Learning: Every User Makes It Smarter
Once continual learning arrives, AI improvement will stop being driven by pre-training runs. Instead, every user interaction globally feeds back into the model. It's a network effect applied to intelligence — and it's fundamentally different from anything that exists today.
Snapshots ()
Key Quotes ()
This episode
Cast
-
Cited for a quote about context-length degradation during training vs. serving, interpreted as a hint that short-horizon RL may not generalize.
-
Cited as a benchmark for domain expertise in space launch business building, used to question the limits of RLVR generalization.
-
Used as a benchmark for political intelligence, testing whether an RLVR-trained AI could match LBJ's Senate campaign strategy in Texas 1948.
-
Used as a benchmark for business-building capability, questioning whether a continually-learning AI could match Walmart's founder.
-
Mentioned as the collaborator for an impromptu blackboard lecture on on-policy self-distillation linked in the episode description.
-
Mentioned humorously as the reason you can't run 1,000 AI agents through Amazon's checkout flow — he'd detect and shut down the bots.
-
Sponsor of the episode; a fintech banking platform that automates invoice processing and bill pay for businesses.
-
Track
Used as an example of why parallel agent rollouts on live websites are not feasible — Amazon would detect and block bot traffic.
-
Discussed via Dario Amodei's quote about context-length degradation, used as evidence that RLVR generalization may have limits.
-
Mentioned as the organization that released AlphaZero, the predecessor to EfficientZero discussed as a dreaming analogy.
-
Used as a hypothetical benchmark for AI general capability — could a sufficiently RLVR-trained AI have built SpaceX given $100M in 2002?
-
Cited as an example of a live online-learning product (CursorTab), which learns from over 400 million daily edit acceptance signals.
-
A data-efficient game-playing model that plays dozens of internal simulated games per real step; used as an analogy for AI 'dreaming'.
-
DeepMind's game-playing AI, mentioned as the precursor to EfficientZero in the context of sample efficiency research.
-
Mentioned alongside Cursor and Claude as tools with a '/compact' feature representing a lightweight simulacrum of continual learning.
Stats
This episode
Claims & Sources
2 / 12 cited (17%)
Factual claims made this episode, and whether a source was named.
AI models are approximately 1/1,000,000th as sample efficient as humans during training.
Around 30 to 50% of an AI lab's compute budget goes to inference, currently without producing any model improvement.
The CursorTab model online-learns by predicting which code edits get accepted by users, processing over 400 million requests per day.
Short-horizon RL training does not necessarily generalize to long-horizon RL performance, as suggested by context-length degradation in deployed models.
People often report that employees are not net productive until 6 months or more into a job, indicating that on-the-job learning is necessary for competence.
Very few model parameters are actually changed during a single RL training step, concentrating the update only on what is relevant to achieving the outcome.
EfficientZero, a data-efficient game-playing model developed after AlphaZero, could beat a novice human at an unseen Atari game given only 2 hours of play, by internally simulating dozens of games per real step.
Training AI at a short context length and then serving at a longer context length can lead to performance degradation.
Some autistic savant individuals can recall random tables of numbers or nonsense syllables years later with extremely high fidelity, but this volume of recall impairs their ability to understand abstractions and metaphors.
OPSD provides a denser supervision signal than naive RL by training on per-token probability discrepancies between a teacher and student model, rather than projecting a single reward through the whole trajectory.
In a previous post, Dwarkesh argued that RL learns much less information per sample than supervised learning.
Dreaming — where AI models generate and train against self-created simulated environments — could become a fourth axis of scaling alongside pre-training, RL, and inference-time compute.
