No. 002 1 min read
Confidently wrong: how often do language models make things up?
A hallucination benchmark across eight frontier models, 1,200 questions, and one uncomfortable chart.
The question isn't whether models hallucinate—they all do—but how the failure is distributed. Some models fabricate rarely but grandly; others hedge on everything.
Claude Opus 4.5
6.1%
Gemini 3 Pro
8.4%
GPT-5
9.8%
Qwen3 Max
12.7%
Grok 4
14.2%
Mistral Large 3
15.1%
DeepSeek V3.2
16.5%
Llama 4 Maverick
18.9%