Hugging Face Reproduced 2,226 ICML 2026 Papers in 19 Days: 23% of Examined Papers Had a Claim Falsified or Contested

Hugging Face published the results of an open reproduction challenge that ran from July 15 to August 2, 2026, reporting that 1,221 participants attempted 2,226 of the 6,352 papers accepted at ICML 2026 and issued verdicts on 35,908 individual claims. Of the examined papers, 51% (1,103) had at least one claim independently verified, and 23% (496) had at least one claim falsified or contested. 266 papers were fully reproduced, 49 had every claim falsified, and 242 drew opposite verdicts from independent teams. ASAP works only from figures verifiable in Hugging Face's own results report to show what this challenge actually measured and which traps appear when those numbers are quoted.

6,816 logbooks were published in 19 days

Hugging Face's published tally is 1,221 participants, 6,816 public reproduction logbooks, and 2,226 papers attempted, which is 34% of the entire conference. Verdicts were issued on 35,908 individual claims and frozen in a public dataset at challenge close. Participants launched 2,962 Hugging Face Jobs, and 274 of the resulting datasets include full agent traces.

Dividing by time and headcount shows the density. 6,816 logbooks over 19 days is 359 per day, or 5.6 per participant. Judged claims average 16.1 per paper, and 3,978 claims were confirmed experimentally, which is 11.1% of all verdicts. The premise behind every one of these figures is a design that scores individual claims rather than labeling a whole paper pass or fail.

The conference context matters too. ICML 2026 accepted 6,352 papers out of 23,918 submissions, an acceptance rate of 26.6%, and Hugging Face notes submissions roughly doubled from the previous year. In a conference where submissions doubled while reviewer capacity did not, this challenge occupies the position of post-acceptance verification carried out by the community rather than by the review process.

Verdicts came from an open-weights judge model, not from participants themselves

Hugging Face ran an automated Logbook Judge on an open-weights model, GLM-5.2, which re-read every logbook and issued a per-claim verdict from four labels: verified, falsified, toy (evidence at reduced scale), or inconclusive. Self-assessment was explicitly treated as untrusted. Participants picked a paper, extracted its core claims, ran reproduction experiments, and published a Trackio logbook, a static Hugging Face Space holding the write-up, code, artifacts, and optionally the full agent trace.

Execution ran on coding agents including Claude Code, Codex, Cursor, Pi, and OpenResearch's orx, with $20 of compute credit per participant. Those two figures indicate how far the cost of re-running a paper has fallen relative to the era when the same work was done entirely by hand.

The failure side breaks down by label as well. 502 papers produced only toy-scale evidence, and 280 established no results at all, with missing artifacts as the most common cause. A substantial share of reproduction failures therefore reflects code and data that were never published rather than claims that turned out to be wrong, and those are different problems.

The four highlighted falsifications failed in four different ways

The representative falsifications Hugging Face highlighted are spread across theory, implementation, and evaluation design. A paper on optimal robustness in learning-augmented paging claimed robustness of H_k + O(1), while reproduction measured the growth as 0.38 ln k at nine sigma; the true bound is H_k + Θ(log k).

A paper connecting attention's forward pass to Frank-Wolfe had its theorem violated at steps 224, 3,800, and 6,416 in independent tests, with the counterexample constructed in exact rational arithmetic. A paper arguing that self-distillation enables continual learning analyzed reverse KL divergence in theory while the released code used forward KL, and its +4pp result did not reproduce even under the authors' own implementation.

In a paper asking whether transformers need three projections, roughly 66% of evaluated label positions were EOS padding tokens. Recomputed without the padding, the 3.1% quality cost reported in the paper becomes roughly 9.4%. Placed side by side, the four cases are not fabrication but mismatches between theory and implementation, contamination in the evaluation set, and generalizations that missed scale dependence. None of them is visible from reading the paper; all of them require running the code.

Reading 23% as "one paper in four is wrong" is a mistake

The denominator behind 23% is examined papers, not the conference, and dividing 496 by the 6,352 accepted papers gives 7.8%. By the same arithmetic, the 266 fully reproduced papers are 4.2% of accepted papers. With roughly two thirds of the conference never attempted, neither ratio can stand in for the quality of the field.

The composition inside those 496 papers is not uniform either. 49 had every claim falsified, while 242 drew opposite verdicts from different teams. A contested verdict is closer to unresolved than to wrong: reproduction conditions may have differed, scale may have differed, or one of the two teams may be mistaken, and this tally does not separate those cases. The 51% figure on the other side carries the same caveat, since its bar is at least one verified claim rather than a paper that holds up entirely.

The final trap is that the sample is not random. Participants chose their own papers. If selection concentrated on papers with well-packaged code and on papers whose results looked unusually strong, both the verified and the falsified ratios come out higher than the underlying rate. Hugging Face itself writes that reproducibility is not binary, it is adversarial, and extending these ratios to the whole conference discards exactly that point.

What is new is not reproduction but its unit cost

Reproduction studies are not new, but covering 2,226 papers in 19 days is only possible under conditions of $20 credits and coding agents. At the unit cost of a human setting up an environment and running experiments by hand, no equivalent volume was reachable in that window. The largest change in this result is less that four specific papers were falsified and more that the cost of falsifying one fell from days to hours and from hundreds of dollars to tens.

That condition changes the timeline after publication. When hundreds of people re-run a claim in their own environments within days of release, verification stops being a single gate at review time and becomes a process that continues after publication. Hugging Face reports that authors responded positively and that multiple arXiv corrections are in flight, which records a path where falsification leads to revision rather than to sanction.

Hugging Face is equally explicit that humans remain necessary: pure agent execution hits real limits, agents misread scale-dependent behavior, and some evaluation stays irreducibly human for now, with perceptual image quality assessment given as the example. The division of labor this challenge demonstrates puts humans on environment setup and question design while agents carry execution. As the cost of running falls, the bottleneck moves toward deciding what to ask.

Who judges the judge model

Every ratio in this tally is a count of verdicts issued by the GLM-5.2 judge, so the judge's error rate is the error bar on the entire result. Hugging Face removed one bias by discarding self-assessment and delegating verdicts to a separate model, and in exchange introduced judgment quality as a new variable. How many of the 496 falsifications are misreadings, and how many of the 242 contested papers reflect judge noise rather than genuine experimental disagreement, cannot be separated from the published tally alone.

The material for checking is available. Every logbook and verdict is public, 274 full agent traces are published, and verdicts were frozen into a public dataset at close. A follow-up study auditing the judge is therefore possible from the same inputs, and that structure is likely to outlast the specific percentages.

For practitioners the takeaway is concrete. Teams that cite paper benchmarks as adoption evidence now have an extra step: acceptance is not verification, so check whether a logbook exists for the paper being cited. The two failure types that appeared here, theory and released code implementing different things and evaluation sets padded with EOS tokens, are precisely the ones that reading the paper will not catch.

Source: Hugging Face's official blog report on the ICML 2026 open reproductions challenge (August 13, 2026) and its public verdict dataset, compiled by ASAP