Quote · Modern Wisdom
Why The AI Doomers Might Be Right - Robert Wright - #1122
Where this was said
Do AI Systems Actually Understand? The Chinese Room Revisited
At 1:02:20 · chapter starts 1:00:40
Chris asks whether AI systems actually know what they are doing or merely simulate knowing. Wright addresses this through John Searle's Chinese Room thought experiment: a man in a room follows rules to respond to Chinese characters without understanding Chinese — suggesting that a computer program, however fluent in output, has no genuine comprehension. Wright finds two problems with the argument. First, Searle was imagining a deterministic rule-following program, not a deep learning system that generates rich internal representations of meaning through statistical training on vast data. Second, if Searle meant that understanding requires consciousness — subjective experience — then we cannot resolve the question, because consciousness can never be verified in any external entity (Wright cannot be 100% sure even Chris Williamson is conscious). Wright proposes a functional alternative: does the system process information with mechanisms functionally analogous to those at work when humans experience understanding? If so, he is willing to call that understanding in a meaningful sense. He concludes that current AI systems have some but not all the elements of understanding, and he sees no principled barrier to them having all of them eventually.
Philosopher John Searle argued that no computer program could ever truly understand language — it just manipulates symbols without meaning. But Searle was imagining a deterministic program, not a deep learning system that independently develops rich semantic representations. Wright argues the empirical evidence now flatly contradicts Searle's core claim.
