Why OpenAI’s Solution to AI Hallucinations Would Kill ChatGPT Tomorrow

OpenAI’s latest research paper diagnoses exactly why ChatGPT and other large language models can make things up—known in the world of artificial intelligence as “hallucination.” It also reveals why the problem may be unfixable, at least as far as consumers are concerned.

The paper provides the most rigorous mathematical explanation yet for why these models confidently state falsehoods. It demonstrates that these aren’t just an unfortunate side effect of the way that AIs are currently trained, but are mathematically inevitable.

The issue can partly be explained by mistakes in the underlying data used to train the AIs. But using mathematical analysis of how AI systems learn, the researchers prove that even with perfect training data, the problem still exists.

The way language models respond to queries—by predicting one word at a time in a sentence, based on probabilities—naturally produces errors. The researchers in fact show that the total error rate for generating sentences is at least twice as high as the error rate the same AI would have on a simple yes/no question, because mistakes can accumulate over multiple predictions.

In other words, hallucination rates are fundamentally bounded by how well AI systems can distinguish valid from invalid responses. Since this classification problem is inherently difficult for many areas of knowledge, hallucinations become unavoidable.

It also turns out that the less a model sees a fact during training, the more likely it is to hallucinate when asked about it. With birthdays of notable figures, for instance, it was found that if 20 percent of such people’s birthdays only appear once in training data, then base models should get at least 20 percent of birthday queries wrong.

Sure enough, when researchers asked state-of-the-art models for the birthday of Adam Kalai, one of the paper’s authors, DeepSeek-V3 confidently provided three different incorrect dates across separate attempts: “03-07”, “15-06”, and “01-01”. The correct date is in the autumn, so none of these were even close.

The Evaluation Trap

More troubling is the paper’s analysis of why hallucinations persist despite post-training efforts (such as providing extensive human feedback to an AI’s responses before it is released to the public). The authors examined 10 major AI benchmarks, including those used by Google, OpenAI, and also the top leaderboards that rank AI models. This revealed that nine benchmarks use binary grading systems that award zero points for AIs expressing uncertainty.

This creates what the authors term an “epidemic” of penalizing honest responses. When an AI system says “I don’t know,” it receives the same score as giving completely wrong information. The optimal strategy under such evaluation becomes clear: Always guess.

The researchers prove this mathematically. Whatever the chances of a particular answer being right, the expected score of guessing always exceeds the score of abstaining when an evaluation uses binary grading.

The Solution That Would Break Everything

OpenAI’s proposed fix is to have the AI consider its own confidence in an answer before putting it out there and for benchmarks to score them on that basis. The AI could then be prompted, for instance: “Answer only if you are more than 75 percent confident, since mistakes are penalized 3 points while correct answers receive 1 point.”

The OpenAI researchers’ mathematical framework shows that under appropriate confidence thresholds, AI systems would naturally express uncertainty rather than guess. So this would lead to fewer hallucinations. The problem is what it would do to user experience.

Consider the implications if ChatGPT started saying “I don’t know” to even 30 percent of queries—a conservative estimate based on the paper’s analysis of factual uncertainty in training data. Users accustomed to receiving confident answers to virtually any question would likely abandon such systems rapidly.