A 27B model beat Claude Opus 4.8 at replicating papers: Faraday and the 310-task Replica benchmark

Inherent Labs posted arXiv paper 2608.13331 on August 13, 2026, reporting that Faraday, a 27B-parameter agent, outperformed Claude Opus 4.8 and GPT-5.5 on the task of replicating figures from research papers. The Replica benchmark consists of 310 figure-replication tasks drawn from 100 machine learning and AI-for-science papers spanning 1990 to 2026, split into 242 training tasks and 68 test tasks. Faraday beat both baselines on 73 percent of in-distribution tasks and 60 percent of out-of-distribution tasks, averaging a 6 percent improvement over Claude and 8 percent over Codex on the test split. ASAP summarizes what this result measured and what it did not, using the paper itself and the Inherent Labs research page as primary sources.

How replication was turned into a benchmark

Replica is a collection of 310 tasks, each asking an agent to reproduce a figure from a paper without access to the original plot. The sources are 100 machine learning and AI-for-science papers published between 1990 and 2026. The tasks split into 242 for training and 68 for testing, and the test split includes domains held out from training.

Scoring runs through an automated rubric. Each criterion receives a continuous score between 0 and 1, and the per-dimension scores are averaged into an overall score for the rollout. The authors measured how closely that judge tracks human experts using Kendall τ, reporting 0.19 for the rubric judge against 0.15 for a baseline judge. On noise, eight baseline judge samples were required to reach the level obtained with three rubric judge samples.

The execution budget is part of the benchmark. Each task gets a 60-minute time limit and a single one-seventh MIG slice of an H200 GPU. The paper runs 47 pages with 12 figures and is authored by Damon Falck, Samer Sabri, Anja Surina, Thom Foster, Anya Sims, Sam Devlin, Dylan Rogers, Tantum Collins, Kaloyan Aleksiev, Louis Kirsch and Edward Hughes.

How a 27B model went up against frontier systems

Faraday is a 27B-parameter agent post-trained from Qwen3.6-27B using a modified version of GRPO, with LoRA fine-tuning at rank 128 and α=128. The full weights were never retrained; adapters carried the long-horizon reinforcement learning.

The comparison conditions are stated explicitly in the paper. Claude Opus 4.8 ran in the Claude Code harness and GPT-5.5 in the Codex harness, both at extra-high thinking effort. A GLM-5.2 baseline ran at max thinking effort in the Claude Code harness, described as the best reported harness for TerminalBench. For Faraday the authors pinned the thinking effort of its own Codex tool to extra-high to keep the comparison fair. The paper names Claude Opus 4.8 as its strongest baseline.

Three numbers carry the result. In distribution, Faraday outperformed both Claude and Codex on 73 percent of tasks. Out of distribution, it did so on 60 percent. Averaged over the test split, it improved 6 percent over Claude and 8 percent over Codex.

What this win proves, and what it does not

The winning system is a 27B model named Faraday that calls a coding agent as its tool, and the thing running inside that tool is Codex at extra-high thinking effort. What the experiment compared is therefore not 27B weights against frontier weights, but a coding agent used directly against a coding agent with a supervisory layer on top.

Miss that distinction and the conclusion inflates. The finding is not that 27B is smarter than Opus 4.8. It is that on this particular long-horizon task, a thin layer deciding what to attempt and what to abandon mattered more than model size. The authors' own framing, describing Faraday as adding a layer of scientific intuition, points at the same place.

For the same reason the result reads more naturally as evidence about architecture than about scale. Margins of 6 and 8 percent are not a gap that displaces frontier models. What makes them interesting is the budget they came from: 60 minutes and one seventh of an H200. If a supervisory layer trained under that budget produces this difference, the cost-effectiveness of training that layer compares favorably against adding parameters.

Is redrawing a figure the same as replicating a result

Replica's 310 tasks measure a proxy, and the authors name the gap themselves: the paper states that perfectly reproducing a plot is not the same as a successful replication, because a graph of the same shape carries no guarantee that the procedure behind it matched the original.

The limitation opens in both directions. One side holds cases where a flawed procedure happened to produce a similar figure; the other holds cases where the procedure was sound but the figure diverged. The authors report instances of the second kind directly, noting that Faraday failed in several cases where they are confident the original result was obtained rigorously.

Judge accuracy compounds this. The rubric judge tracking humans more closely than the baseline judge is a relative claim, and a Kendall τ of 0.19 in absolute terms sits well short of matching expert rankings. When an automatically scored benchmark separates models by 6 to 8 percent, part of that margin plausibly falls inside the judge's noise band. The authors reporting how many samples are needed to suppress that noise suggests the design anticipated the objection.

Replication as a training signal

Replication is a usable reinforcement learning reward because the answer already sits inside the paper, which is what let 100 papers yield 310 Replica tasks. The original figure becomes the grading target without any new human labeling, and the number of papers directly determines the number of tasks.

Its position in the research-automation debate follows from that. Making a system produce novel discoveries is hard to grade; making it reproduce a published result comes with grading attached. Teaching replication before attempting discovery is a defensible ordering from a reward-design standpoint.

The ordering brings a structural ceiling with it. How far capability learned from replication transfers to genuinely new problems is a separate question, and 73 percent in distribution falling to 60 percent out of distribution is the first signal that transfer is not unbounded. Spanning papers from 1990 to 2026 looks like an attempt to reduce era bias, but how stable a 60 percent figure is on a 68-task test split needs further verification.

What research teams can take from this

For university and corporate labs, the practical value here sits in Replica's construction more than in Faraday itself. Pulling figure-replication tasks from papers in your own field to build an internal evaluation set requires no special infrastructure, and the property that the grading target ships inside the paper carries over intact.

The budget conditions are realistic too. A 60-minute limit and a one-seventh H200 slice per task means a single GPU can run several tasks in parallel. Compared with calling frontier model APIs at volume, the unit cost of repeated experimentation is very different.

One thing to check before copying the setup. Faraday's performance was measured on one specific base model, Qwen3.6-27B, in combination with one specific tool, Codex. Whether the same margin survives a different base model or a different coding agent is not answered in the paper, and it is the first variable worth testing in-house.

What to verify next

The largest gap the paper leaves is independent reproduction. The first thing to check is how far 73 percent and 60 percent move when the Replica tasks, the scoring rubric and the harness combination run in another team's hands. Agentic benchmarks shift substantially with the harness, and this experiment places different harnesses side by side by design.

The second is a taxonomy of the failures. If, as the authors say, replication failed on papers whose original results were rigorous, then separating model limitations from task-design limitations determines the accuracy of the next version. Should this benchmark start being used to cast replication failure as a problem with the original paper, that usage is unsafe until the separation is done.

Source: Damon Falck et al., "Training AI Scientists to Replicate Research" (arXiv:2608.13331, August 13, 2026) and the Inherent Labs research page (August 14, 2026). Summarized by ASAP.