By 2027, AI models may improve faster from real-world deployment experience than from any pre-training run — and that shift will be both scary and transformative.
Dwarkesh Podcast
By 2027, AI models may improve faster from real-world deployment experience than from any pre-training run — and that shift will be both scary and transformative.
TL;DR
Dwarkesh Patel argues that the next frontier in AI isn't just scaling RLVR training on verifiable tasks, but enabling AIs to learn continuously on the job from real-world deployment [1] — Dwarkesh Patel "The labs' core AGI bet is that training on enough verifiable, containerized RL environments will produce agents capable of open-ended probl…" . He identifies "grindability" — the ability to run parallel rollouts in deterministic simulators — as an underrated bottleneck, explaining why computer use lags behind coding [2] — Dwarkesh Patel "Many skills humans have — winning court cases, building businesses, day trading — require real-world interaction with sparse, delayed feedb…" 05:30 . Two speculative solutions emerge: on-policy self-distillation (OPSD) to compress session learnings back into model weights [3] — Dwarkesh Patel "OPSD trains the base model to match what a veteran 'teacher' model — loaded with a full session's context — would predict. It needs no oute…" 11:05 , and "dreaming," where AIs build their own simulated environments to rehearse skills [4] — Dwarkesh Patel "Dreaming means the model spends compute generating its own simulated RL environments, then trains against them — rehearsing skills relevant…" 14:00 . The key takeaway: by 2027–28, AI improvement may come primarily from deployment experience, not pre-training.
Dwarkesh Patel argues that the next frontier in AI is enabling models to learn continuously from real-world deployment — and explores grindability, on-policy self-distillation, and 'dreaming' as the keys to getting there.
Dwarkesh outlines the core labs' bet: scale verifiable RL training across millions of tasks to produce a general problem-solving agent. He also raises the counterargument that AI is 1/1,000,000th as sample efficient as humans [1] — Dwarkesh Patel "1/1,000,000th sample efficiency: Current AI models are one-millionth as sample efficient as humans during training, a deficit Dwarkesh prev…" 00:42 .
Dwarkesh introduces 'grindability' — the ability to run parallel rollouts in deterministic simulators — as the underrated reason computer use AI lags far behind coding, despite being equally verifiable [1] — Dwarkesh Patel "Verifiability alone doesn't explain which domains see rapid AI progress. The real bottleneck is grindability: can you run thousands of para…" 02:12 .
Dwarkesh questions whether RLVR generalizes from containerized tasks to complex real-world domains like politics or business, citing Dario Amodei's context-length degradation comment as a key data point [1] — Dwarkesh Patel "Dario Amodei noted that training at short context lengths and serving at long ones can cause performance degradation. Dwarkesh reads this a…" 06:55 .
Dwarkesh explores on-policy self-distillation (OPSD) as the leading technique for continual learning, contrasting it with naive SFT and explaining why RL's sparse parameter updates are a feature, not a bug [1] — Dwarkesh Patel "OPSD trains the base model to match what a veteran 'teacher' model — loaded with a full session's context — would predict. It needs no oute…" 11:05 .
Dwarkesh introduces 'dreaming' — AI building its own RL environments to rehearse skills — using EfficientZero as an analogy, and positions it as a potential fourth scaling axis alongside pre-training, RL, and inference compute [1] — Dwarkesh Patel "Dreaming means the model spends compute generating its own simulated RL environments, then trains against them — rehearsing skills relevant…" 14:00 .
Dwarkesh sketches a concrete 2027–28 scenario where RLVR-primed agents get week-long context windows, do real work, and distill session learnings back into weights — enabling AI to improve primarily from deployment experience [1] — Dwarkesh Patel "RLVR produces an agent competent enough to deploy. That agent gets a week-long context window and does real work. At the end of the week, a…" 16:30 .
Chapter 1 · 00:00
Dwarkesh outlines the core labs' bet: scale verifiable RL training across millions of tasks to produce a general problem-solving agent. He also raises the counterargument that AI is 1/1,000,000th as sample efficient as humans [1] — Dwarkesh Patel "1/1,000,000th sample efficiency: Current AI models are one-millionth as sample efficient as humans during training, a deficit Dwarkesh prev…" 00:42 .
The labs' core AGI bet is that training on enough verifiable, containerized RL environments will produce agents capable of open-ended problem-solving for weeks on end. Skeptics argue the fundamental deficits — data inefficiency, no continual learning — will simply be steamrolled by scale, just as compute steamrolled NLP.
Current AI models are one-millionth as sample efficient as humans during training, a deficit Dwarkesh previously explored in depth.
People often say employees aren't net productive until 6+ months on the job, illustrating why continual learning matters for competence.
Chapter 2 · 02:12
Dwarkesh introduces 'grindability' — the ability to run parallel rollouts in deterministic simulators — as the underrated reason computer use AI lags far behind coding, despite being equally verifiable [1] — Dwarkesh Patel "Verifiability alone doesn't explain which domains see rapid AI progress. The real bottleneck is grindability: can you run thousands of para…" 02:12 .
Verifiability alone doesn't explain which domains see rapid AI progress. The real bottleneck is grindability: can you run thousands of parallel rollouts from identical starting points in a deterministic simulator? Coding can; live websites cannot.
Computer use has made far slower progress than coding and math despite being clearly verifiable, because it lacks grindable, deterministic training environments.
It's not enough for a domain to be verifiable — it must also be 'grindable', meaning you can run thousands of parallel rollouts from identical starting points in a deterministic simulator.
Many skills humans have — winning court cases, building businesses, day trading — require real-world interaction with sparse, delayed feedback. There's no containerized RL environment for 'build a company as well as Sam Walton.' These domains will resist the current training paradigm.
Chapter 3 · 06:10
Dwarkesh questions whether RLVR generalizes from containerized tasks to complex real-world domains like politics or business, citing Dario Amodei's context-length degradation comment as a key data point [1] — Dwarkesh Patel "Dario Amodei noted that training at short context lengths and serving at long ones can cause performance degradation. Dwarkesh reads this a…" 06:55 .
Dario Amodei noted that training at short context lengths and serving at long ones can cause performance degradation. Dwarkesh reads this as evidence that RLVR generalization is not infinite — a crack in the foundation of the labs' AGI bet.
Labs burn 30–50% of compute on inference — and currently gain zero improvement to the model from it. Meanwhile, deployment is where the most valuable learning signals actually exist: real organizational context, real mistakes, real feedback.
Labs spend 30 to 50% of their total compute on inference, yet this compute currently plays no productive role in improving the model.
Chapter 4 · 08:41
Dwarkesh explores on-policy self-distillation (OPSD) as the leading technique for continual learning, contrasting it with naive SFT and explaining why RL's sparse parameter updates are a feature, not a bug [1] — Dwarkesh Patel "OPSD trains the base model to match what a veteran 'teacher' model — loaded with a full session's context — would predict. It needs no oute…" 11:05 .
The CursorTab model online-learns by predicting which edits users accept, processing over 400 million requests per day.
OPSD trains the base model to match what a veteran 'teacher' model — loaded with a full session's context — would predict. It needs no outer-loop reward, provides denser per-token supervision than RL, and avoids overwriting existing knowledge. It's the most credible path to continual learning today.
Unlike RL-VR, on-policy self-distillation doesn't require an outer-loop verifiable reward — it only requires a model that can learn the right things within the context window.
On-policy self-distillation provides per-token supervision rather than a single end-of-trajectory reward, making it a much denser training signal than naive RL.
Naively fine-tuning on session transcripts trains models to recall everything with perfect fidelity — but that's not how skill acquisition works. RL concentrates updates only where they matter and changes very few parameters, which is exactly what you want when you don't want to forget what you already know.
RL training changes very few model parameters per step, concentrating updates only on what is relevant to achieving the outcome — a crucial property for continual learning.
Dreaming means the model spends compute generating its own simulated RL environments, then trains against them — rehearsing skills relevant to the actual user's real-world context. Like EfficientZero playing dozens of internal games per real step, this could make AI far more sample-efficient without requiring external data.
EfficientZero, trained to be data-efficient, could beat a novice human at an unseen Atari game given only 2 hours of play time.
Chapter 5 · 15:23
Dwarkesh introduces 'dreaming' — AI building its own RL environments to rehearse skills — using EfficientZero as an analogy, and positions it as a potential fourth scaling axis alongside pre-training, RL, and inference compute [1] — Dwarkesh Patel "Dreaming means the model spends compute generating its own simulated RL environments, then trains against them — rehearsing skills relevant…" 14:00 .
Dreaming — where a model generates and trains against its own simulated environments — could become a fourth scaling axis alongside pre-training, RL, and inference-time compute.
RLVR produces an agent competent enough to deploy. That agent gets a week-long context window and does real work. At the end of the week, a thumbs up triggers OPSD or dreaming to distill everything it learned back into the base weights. Repeat. The AI's skills start expanding beyond its original training domains — with each cycle.
Dwarkesh envisions that by 2027–28, effective context lengths will expand enough for AIs to co-work with a human for a full week of wall-clock time.
Chapter 6 · 17:23
Dwarkesh sketches a concrete 2027–28 scenario where RLVR-primed agents get week-long context windows, do real work, and distill session learnings back into weights — enabling AI to improve primarily from deployment experience [1] — Dwarkesh Patel "RLVR produces an agent competent enough to deploy. That agent gets a week-long context window and does real work. At the end of the week, a…" 16:30 .
Once continual learning arrives, AI improvement will stop being driven by pre-training runs. Instead, every user interaction globally feeds back into the model. It's a network effect applied to intelligence — and it's fundamentally different from anything that exists today.
No indexed bits in this chapter.
Verifiability alone doesn't explain which domains see rapid AI progress. The real bottleneck is grindability: can you run thousands of parallel rollouts from identical starting points in a deterministic simulator? Coding can; live websites cannot.
Once continual learning arrives, AI improvement will stop being driven by pre-training runs. Instead, every user interaction globally feeds back into the model. It's a network effect applied to intelligence — and it's fundamentally different from anything that exists today.
This episode
Factual claims made this episode, and whether a source was named.
AI models are approximately 1/1,000,000th as sample efficient as humans during training.
Around 30 to 50% of an AI lab's compute budget goes to inference, currently without producing any model improvement.
The CursorTab model online-learns by predicting which code edits get accepted by users, processing over 400 million requests per day.
Short-horizon RL training does not necessarily generalize to long-horizon RL performance, as suggested by context-length degradation in deployed models.
People often report that employees are not net productive until 6 months or more into a job, indicating that on-the-job learning is necessary for competence.
Very few model parameters are actually changed during a single RL training step, concentrating the update only on what is relevant to achieving the outcome.
EfficientZero, a data-efficient game-playing model developed after AlphaZero, could beat a novice human at an unseen Atari game given only 2 hours of play, by internally simulating dozens of games per real step.
Training AI at a short context length and then serving at a longer context length can lead to performance degradation.
Some autistic savant individuals can recall random tables of numbers or nonsense syllables years later with extremely high fidelity, but this volume of recall impairs their ability to understand abstractions and metaphors.
OPSD provides a denser supervision signal than naive RL by training on per-token probability discrepancies between a teacher and student model, rather than projecting a single reward through the whole trajectory.
In a previous post, Dwarkesh argued that RL learns much less information per sample than supervised learning.
Dreaming — where AI models generate and train against self-created simulated environments — could become a fourth axis of scaling alongside pre-training, RL, and inference-time compute.
This episode
Cited for a quote about context-length degradation during training vs. serving, interpreted as a hint that short-horizon RL may not generalize.
Cited as a benchmark for domain expertise in space launch business building, used to question the limits of RLVR generalization.
Used as a benchmark for political intelligence, testing whether an RLVR-trained AI could match LBJ's Senate campaign strategy in Texas 1948.
Used as a benchmark for business-building capability, questioning whether a continually-learning AI could match Walmart's founder.
Mentioned as the collaborator for an impromptu blackboard lecture on on-policy self-distillation linked in the episode description.
Mentioned humorously as the reason you can't run 1,000 AI agents through Amazon's checkout flow — he'd detect and shut down the bots.
Sponsor of the episode; a fintech banking platform that automates invoice processing and bill pay for businesses.
Used as an example of why parallel agent rollouts on live websites are not feasible — Amazon would detect and block bot traffic.
Discussed via Dario Amodei's quote about context-length degradation, used as evidence that RLVR generalization may have limits.
Mentioned as the organization that released AlphaZero, the predecessor to EfficientZero discussed as a dreaming analogy.
Used as a hypothetical benchmark for AI general capability — could a sufficiently RLVR-trained AI have built SpaceX given $100M in 2002?
Cited as an example of a live online-learning product (CursorTab), which learns from over 400 million daily edit acceptance signals.
A data-efficient game-playing model that plays dozens of internal simulated games per real step; used as an analogy for AI 'dreaming'.
DeepMind's game-playing AI, mentioned as the precursor to EfficientZero in the context of sample efficiency research.
Mentioned alongside Cursor and Claude as tools with a '/compact' feature representing a lightweight simulacrum of continual learning.
Stats
We use essential and analytics cookies to run Vuci. To understand how the site is used: Privacy Policy.
Install Vuci on your phone
Add it to your home screen for a faster, app-like experience.
Install Vuci on your phone
Tap the Share button, then “Add to Home Screen”.
A new version is available
Reload to get the latest Vuci.