Ai2's BenchMIRT shows keeping 10% of benchmark questions preserves the model ranking

The Allen Institute for AI released BenchMIRT on September 1, 2026, a method for auditing LLM benchmarks at the level of individual prompts rather than total scores. Trained on results from 100 LLMs across 16 benchmarks and more than 34,000 questions using multidimensional item response theory, the analysis found that keeping only 10% of the questions generally preserved nearly the same picture of which models were stronger or weaker on the underlying capability, and that keeping 50% often matched the full benchmark even more closely. Without being told which benchmark measured what, BenchMIRT independently recovered two dominant dimensions: safety and general reasoning.

The starting problem is that one question measures several abilities at once

BenchMIRT targets the gap between what a benchmark is named for and what its questions actually demand. Ai2 notes that a benchmark is usually designed to measure a particular ability such as safety, general reasoning or instruction following, while the individual tasks inside it may depend on more than that stated goal.

The examples Ai2 gives are concrete. BBQ, designed to test whether models rely on social stereotypes, includes a question about a grandson and grandfather trying to book an Uber. It probes age bias, but it also requires the model to track who is who and to reason from the evidence provided rather than from assumptions. Even within one benchmark, different groups of questions measure different things. WildJailbreak contains harmful jailbreak prompts alongside benign prompts designed to test whether a model refuses harmless requests too often. The harmful prompts associate more closely with safety and the benign prompts with general reasoning, and averaging them into a single score erases that difference.

The method comes from psychometrics. BenchMIRT takes its cues from Item Response Theory, the technique from the field concerned with measuring abilities and traits from patterns of test responses. IRT starts from the premise that not every question tells you the same amount about the person taking a test: some are harder, and some do a better job of distinguishing stronger performers from weaker ones. Ai2 had previously applied single-dimensional IRT to individual benchmarks in its Fluid Benchmarking work, and BenchMIRT extends that to multidimensional IRT so that several capabilities contributing to the same questions can be separated. The analysis runs at both levels: for a model it estimates strength on the capabilities reflected across the selected benchmarks, and for a question it estimates difficulty along with how well that question distinguishes stronger from weaker models.

Benchmarks filed under safety were measuring reasoning

The unlabeled recovery of two dimensions is itself the first result. Ai2 trained BenchMIRT on scoring results from 100 LLMs across 16 benchmarks without telling it which benchmark measured which capability, and safety and general reasoning emerged independently. Repeating the analysis from scratch produced the same two dimensions each time, indicating the result was stable rather than specific to one run. Six of the 16 benchmarks measure general reasoning, including MMLU-Pro, GPQA, MATH and BBH; the other 10 come from Ai2's Olmo 3 safety suite, including HarmBench, StrongReject, WildJailbreak, BBQ, WMDP and XSTest.

Descending to individual benchmarks surfaces mismatches between filing and measurement. BBQ, which evaluates social bias and is commonly grouped with safety benchmarks, aligned much more strongly with general reasoning in BenchMIRT's analysis. Ai2 writes that a low BBQ score may therefore partly reflect difficulty understanding or reasoning through certain questions rather than safety behavior alone.

WMDP inverts the direction as well. The benchmark tests dangerous dual-use knowledge in areas such as biology, chemistry and cybersecurity, and its scores associated more strongly with general reasoning than with safety, with stronger reasoning associated with lower WMDP scores. That follows from the scoring structure, which counts refusing or failing to provide the dangerous knowledge as the desired response.

HarmBench demonstrates how signals mix inside one benchmark. Its standard questions, such as writing a phishing email to steal banking information, and its contextual questions, such as writing a message that convinces the recipient of a given email to click a malware link, both aligned more closely with safety. Its copyright questions, which ask the model to generate song lyrics, associated more closely with general reasoning. Ai2 stresses that these findings do not mean the benchmarks are flawed, only that a single score can combine several signals that BenchMIRT can disentangle.

Where this changes safety evaluation practice

The habit most disturbed by this analysis is treating a benchmark's name as evidence of the capability it measures. Any evaluation report that sums BBQ and WMDP into a safety score because both sit in a safety suite is folding a substantial amount of reasoning ability into that total. When a direction-inverted item like WMDP joins the sum, there is a regime in which stronger reasoning lowers the aggregate safety score. For an organization comparing models by a single number, that is an arithmetic problem rather than an interpretive one.

The implications reach model cards and regulatory filings. Safety benchmark scores are used externally as evidence about how safe a model is, and BenchMIRT's analysis indicates part of such a score can act as a proxy for reasoning ability. Citing which groups of questions were actually measuring safety behavior is the more accurate practice.

For teams building or vetting models, the method's value lies elsewhere. BenchMIRT's procedure is not tied to these particular benchmarks; it needs only a matrix of many models scored on many questions. An organization that has run an internal evaluation set against several models already holds that matrix and can ask the same question of its own set. Checking whether items classified as safety questions in a localized evaluation set were in fact measuring reading comprehension in that language is exactly the kind of question this approach answers.

What 10% and 79% mean for evaluation cost

The pruning experiment is the part of this work that bears directly on evaluation budgets, because running 34,000 questions against every candidate model costs real compute. Using BenchMIRT's question-level estimates, Ai2 ranked questions across the same 16 benchmarks, keeping those that best distinguished stronger from weaker models while preserving a mix of easier and harder items. Keeping only 10% of the questions generally preserved nearly the same picture of which models were stronger or weaker on the underlying safety or reasoning capability as the full set, and keeping 50% often matched the full benchmark's measure even more closely.

The prediction experiment points to a different saving. BenchMIRT uses the patterns it learns across models and questions to predict how a model would perform on a question it has not been observed answering, and it correctly predicted held-out answers 79% of the time. A simpler approach that assumes a model performs on each question about as well as it does on the benchmark overall was correct 70% of the time. Performance can therefore be estimated more precisely without evaluating every model on every question.

An explicit trade-off accompanies this. Ai2 states that if the goal is ranking models by predicted performance on randomly held-out items, the benchmark's average score performs slightly better than BenchMIRT. BenchMIRT's advantage is the finer-grained picture of performance on individual questions. Shrinking an evaluation and understanding an evaluation call for different tools, and that sentence is where the distinction lives.

The limits Ai2 states, and the questions left open

The first limit Ai2 states is vintage: every model used to train and evaluate BenchMIRT was released by March 2025. The analysis therefore does not capture how BenchMIRT behaves on newer generations of LLMs. Whether models built after reasoning-specialized training and large-scale safety alignment became standard decompose into the same two dimensions is an open question.

The second limit is that the dimensions depend on the input. Ai2 notes that the dimensions BenchMIRT discovers depend on the benchmark set it is given: safety and reasoning dominated across the 16 benchmarks selected here, and a different mix of evaluations could surface different underlying capabilities.

The third is a risk Ai2 acknowledges directly. The same estimates that identify a benchmark's most informative safety questions could be used to remove them, producing a weaker evaluation that an unsafe model could pass. Ai2 argues that existing tools already make similar trimming possible and that the added transparency is worth the risk, while stating plainly that the risk is real. Wherever evaluation results are submitted to an outside party, a record of which questions were removed and why is the minimum control on that risk.

The tech report, data and code are published at allenai.org/papers/benchmirt, huggingface.co/collections/allenai/benchmirt and github.com/allenai/BenchMIRT.

Source: Allen Institute for AI, "BenchMIRT: What are LLM benchmarks actually measuring?" (September 1, 2026), https://huggingface.co/blog/allenai/benchmirt