The longer the document, the more AI fabricates: a 172-billion-token hallucination study

AI hallucination is sharper the longer the context grows in document question answering. A March 2026 paper, "How Much Do LLMs Hallucinate in Document Q&A," evaluated 35 open-weight models across 172 billion tokens. At 32K tokens even top models fabricated 5~7%, and at 200K tokens every model exceeded 10%. The key point is that the ability to find facts and the ability not to fabricate are separate. ASAP summarizes the result from the primary source.

From 32K to 200K: the slope of the hallucination curve

The paper found that hallucination rises sharply as context grows longer. At 32K tokens top models fabricated 5~7%, but that nearly tripled at 128K, and at 200K all 35 models exceeded 10%. It is a measurement at the scale of 172 billion tokens.

How you read those numbers matters. Stretching the context roughly sixfold from 32K to 200K widened the fabrication rate along a curve rather than a staircase, which is worth noticing. Even the top models' 5~7% is not a comfortable figure. If five or six out of a hundred answers contain facts absent from the document, the cost of a reviewer hunting down those five or six erodes the benefit of trusting the rest. And the fact that every model without exception crossed 10% at 200K suggests this is not a flaw of one architecture but a structural effect of the long-context condition itself.

Being good at retrieval does not make a model honest

The paper's key finding is that the ability to find evidence and the ability not to fabricate are separate. A model good at locating facts can still generate falsehoods. That means the two abilities must be measured separately.

This separation changes how practitioners should read benchmarks. We tend to lump top-ranked models together as "accurate," but high retrieval accuracy is no guarantee of low fabrication. Fetching an answer and admitting what it does not know are closer to different muscles. When adopting RAG over internal documents or regulations, picking a model on retrieval hit rate alone can miss a dangerous tendency to fabricate.

The illusion that lowering temperature is safer

There is a paradox in the temperature setting too. Temperature 0.0 gave the best accuracy in about 60% of cases, but coherence loss happened 48 times more often than at temperature 1.0. For most models, higher temperature actually reduced fabrication.

Hardware is not an excuse

The results were consistent across all three hardware platforms tested in the study. Hallucination rates were similar on Nvidia H200, AMD MI300X, and Intel Gaudi 3. That means you need not pick specific hardware to reduce hallucination.

So what should you not feed in

The paper shows with numbers that long-document Q&A should not be trusted blindly. The longer the context you stuff in, the more AI fabricates, so feeding only the key evidence, kept short, is safer. In retrieval-augmented generation (RAG), more is not the answer.

Still, flattening this paper's conclusion into "always shorter" would be a mistake. This measurement alone cannot cleanly separate whether the rising fabrication comes from context length itself or from the noise and conflicting information common in long documents. A workable rule in practice is to manage the density of evidence rather than the volume. Trimming irrelevant passages to raise the signal-to-noise ratio, and forcing a cited source for every answer so fabrication can be checked after the fact, is the better path.

Source: ASAP summary of "How Much Do LLMs Hallucinate in Document Q&A Scenarios?" (arXiv 2603.08274, March 9, 2026; JV Roig, 35 open-weight models, 172 billion tokens, top models fabricating 5~7% at 32K and all exceeding 10% at 200K, temperature 0.0 most accurate but 48x more coherence loss, consistent across H200, MI300X, and Gaudi 3).