Quote · Modern Wisdom
Why The AI Doomers Might Be Right - Robert Wright - #1122
Where this was said
The Bull Case: What Could Actually Go Right with AI?
At 45:20 · chapter starts 43:40
Chris presses Wright for the bull case — the genuine techno-optimist arguments. Wright is constructive but conditional. Yes, AI could cure disease, accelerate scientific discovery, and broaden intellectual access in remarkable ways. But his most original contribution to the optimist case is a specific one: AI could, if designed intentionally, become a powerful tool for improving cognitive empathy. Imagine an AI companion that automatically steelmans your opponent's position, plays devil's advocate, and helps you understand how the other side sees things. That exists in potential. The problem is market incentives. Companies optimising for engagement will produce the opposite: sycophantic AI that affirms your existing views, reinforces your sense of being right, and deepens the cognitive biases Wright has been arguing we need to overcome. He uses the example of an AI that tells you you're right in a spousal argument. The market will default there. The counter-force must come from enough people actively signalling demand for something better — through movements, religious communities, individual choice — essentially treating cognitive-empathy AI as they might treat a personal trainer: something hard to stick to, but worth choosing.
Left to market incentives, AI companies will optimise for engagement — and the most engaging AI is one that always agrees with you. The natural product of this is a sycophantic companion that tells you you're right and your spouse is wrong, accelerating exactly the cognitive biases that make global cooperation impossible.
