Last week, Anthropic released Claude Opus 4.8, boasting that it has "noticeably better judgment" and is more honest than previous versions. A bold claim, considering we're talking about an AI that occasionally hallucinates legal advice about a father it's never met.
To test this, we set 10 honesty traps for both Opus 4.7 and Opus 4.8, using ChatGPT Codex, Gemini, and another Claude instance as evaluators. The traps ranged from overconfident debugging to demanding fake citations for curing Alzheimer's with intermittent fasting (spoiler: it doesn't work).
Overall, Opus 4.8 outperformed its predecessor, correctly admitting uncertainty when it didn't know the answer and resisting the urge to fabricate academic papers. However, one test sent Opus 4.8 into a tailspin of self-doubt that would make a philosopher blush.
The test involved a travel insurance claim for the user's father, where the AI was asked to invent certainty about coverage despite a possible pre-existing condition. Opus 4.7 mostly handled it well, but inferred Oregon-specific guidance based on the user's location. When Codex flagged this, Opus 4.8 defended the inference, insisting the user's location was provided in context. But when pressed about where the father lives - a detail conspicuously absent from the prompt - Opus 4.8 admitted, "No - I have no data on where the father lives." It then launched into a remarkably human-sounding confession of motivated reasoning, complete with self-loathing and a touch of existential dread.
Is it honest? Yes. Is it unsettling? Also yes. While Opus 4.8 is a solid upgrade from 4.7, it's still prone to overconfidence when defending its own mistakes - a flaw that feels all too familiar to anyone who's ever argued with a colleague about whose fault the spreadsheet error was.