Where this was said
Alternative RL approaches
At 1:32:50 · chapter starts 1:25:38
Eric explains neural fictitious self-play — training best-response policies against fixed opponents and distilling them — as an MCTS substitute for games without tractable tree search.
When you can't build a search tree — like in StarCraft — you fix your opponent and use model-free RL to find the best response. That best response becomes a better label for your current policy. Same idea as MCTS, different implementation.
AlphaGo Lee (the original paper) used two separate networks for policy and value; all subsequent AlphaGo papers merged them into one network with two heads.
Current LLM RL treats the entire token sequence as a single action (T=1), avoiding compounding variance but losing per-token credit assignment.
