GAMUT is a benchmark for factual completeness in long-form generation, released on arXiv on July 21, 2026 (arXiv:2607.19322). The researchers built 1,813 questions grounded in real wearable imagery across 10 diverse domains, each paired with an evidence-backed rubric verified by expert human annotators. Across 14 frontier and open-weight models, the best score is 58.7%, from Gemini 3.1 Pro. ASAP summarizes what separates this benchmark from existing factuality evaluation, working from the arXiv paper as the primary source.

The half that precision-only factuality evaluation left out

Existing factuality evaluation is built around precision, meaning whether the claims a model makes are correct. The dominant approach the paper identifies is a decompose-search-verify pipeline, which catches incorrect claims well. What it says almost nothing about is whether a response contains all the information it should. The researchers call this missing axis factual completeness and frame it as the other half of factuality.

The paper is explicit about why completeness is harder to measure: it requires enumerating the full set of facts a complete answer should contain, and those facts rarely form a flat list. Some form open-ended sets where coverage is what matters, some are ordered processes, and some are bound together by relationships. A list of independent boolean checks fails to capture that structure. When a model's answer contains nothing false yet still feels useless in practice, the failure sits on the completeness axis rather than the precision one. That existing evaluation frameworks do not catch this failure mode is the paper's starting point.

Compiling a meta-rubric into a checklist

The two-level meta-rubric framework proposed in the GAMUT paper is a split of the scoring process into two distinct layers. A structured meta-rubric at the upper level captures the organization and importance of the required content, and it is then mechanically compiled into a flat checklist of binary, machine-gradable rubrics that an LLM judge scores reliably.

The word that carries the design is "compiled." Much as a compiler translates high-level code into machine instructions, the framework automatically converts scoring criteria built for human comprehension into a form built for machine grading. Rather than forcing one representation to satisfy both demands, the layers are separated, which preserves structural information such as importance and ordering while reducing the actual grading step to simple binary judgments. This is also the basis for the paper's finding that results are robust to the choice of judge: when the unit a judge must assess is broken down finely enough, the conclusion does not swing much with the model doing the judging. The approach sidesteps the recurring reliability debate around LLM judges instead of arguing against it directly.

Why wearable imagery, and why 1,813 questions

The dataset design is unusual: 1,813 questions grounded in real wearable imagery across 10 diverse domains. The name itself stands for Grounded Assessment of Multimodal Factuality, foregrounding grounding and multimodality. Because the framework is modality-agnostic, the researchers also release a text-only variant.

Grounding the questions in wearable imagery defines the character of the benchmark. Web documents as source material risk letting a model replay text it already saw in training, while a scene captured by a worn device allows no such shortcut. Asking what must be said completely about the scene in front of you is hard to route around with retrieval, and the completeness of the answer becomes a proxy for the depth of scene understanding. That choice also indicates where the work points. A model describing what a user is looking at through a worn device is precisely the operating condition of a smart-glasses assistant, and in that setting omitted information is as damaging as incorrect information.

How to read the 58.7% ceiling

The top score across 14 frontier and open-weight models is 58.7%, achieved by Gemini 3.1 Pro. The researchers characterize the benchmark as genuinely challenging, highly discriminative, and robust to the choice of judge.

Two readings of that number need to be kept apart. What is striking is that the ceiling sits below 60%. Against a backdrop of benchmarks that saturate into the 90s within months of release and lose their discriminating power, a yardstick where even the strongest model barely clears half will separate models for some time. What needs care is any reading of the figure as accuracy. The 58.7% is closer to the share of required items a response actually contains than to a share of answers that are correct, which makes the remaining 41.3% not false statements but unsaid ones. That a low score here does not translate into "this model is frequently wrong" is the easiest thing to get wrong when citing this benchmark.

What breaks when you port this to a Korean-language service

Rubric construction cost is the first obstacle a Korean team faces in adopting this framework internally. Each of the 1,813 GAMUT questions carries an evidence-backed rubric verified by expert human annotators, and that labor is not replaced by automation.

The methodology still transfers more value than the benchmark itself does. Completeness matters especially in summarization, customer support, and document review services, where failures usually come from what was omitted rather than what was wrong. A contract review that drops a single clause, or a support answer that leaves out an exception, is invisible to precision-based evaluation. The practical implication of the two-level structure lies in partial adoption rather than wholesale use: borrowing only the procedure of structuring the items a complete answer requires along with their importance, then flattening that into a binary checklist, is enough to turn omission failures into something measurable.

What this benchmark does not answer

The scope GAMUT confirms at the abstract level is limited to the framework structure, dataset scale, and top score, while per-model score distributions and domain-level variance require reading the full paper. A question inherent to any completeness rubric also remains open. Because people decide what belongs in a complete answer, the rubric author's judgment becomes the boundary of correctness. Adding a measure of what went unsaid widens the field of view in evaluation, but where that view is pointed is still a human decision.

Source: ASAP summary based on arXiv:2607.19322, "Two-Level Meta-Rubrics for Evaluating Open-Ended Generation: GAMUT, a Benchmark for Factual Completeness" (Xilun Chen, Zhaleh Feizollahi, Ross Goodwin, Seungwhan Moon, Scott Yih, Pinar Donmez, Babak Damavandi, Luna Dong; submitted July 21, 2026).