Google DeepMind Runs the First Double-Blind Eval: Gemini 2.5 Flash Lite Measured Inside an H100 Enclave Against a Private Benchmark, at Under 5% Compute Overhead

Google DeepMind published the world's first double-blind evaluation of a proprietary frontier-class model on August 27, 2026. The technical report is defined by four facts: the model under test is Gemini 2.5 Flash Lite, the measurement ran on a Google Cloud Confidential Space a3-highgpu-1g instance with Intel TDX host memory encryption and an NVIDIA H100 80GB Confidential GPU, compute overhead is under 5%, and the participating organizations are the Singapore AI Safety Institute, OpenMined, AVERI, and MLCommons. ASAP reads the pilot for what it actually shipped: not a benchmark score, but the procedure itself as the deliverable.

Benchmark contamination is already a measured problem

The report grounds its problem statement in three prior studies, each carrying a hard number. Singh et al. (2025), "The Leaderboard Illusion," documents one frontier lab testing 27 private model variants on Chatbot Arena and then publishing only the top scorer. Xu et al. (2024) finds strong evidence of benchmark data leakage in both pre-training and post-training for roughly half of the 31 models tested. Schaeffer et al. (2026) quantifies how test set contamination inflates measured performance, with inflation growing alongside both the amount of contamination and model size.

Read together, the three studies separate contamination into two distinct failures. One is deliberate benchmark hacking; the other is accidental leakage of test items into training corpora. Contracts and zero-logging policies address the first reasonably well and the second structurally poorly, because no contract offers a way to prove after the fact that past evaluation prompts did not get swept into the next pre-training crawl. That gap is why the report states the problem cannot be solved with trust and contracts alone.

The more consequential shift is that contamination has moved from suspicion to measurement. Through 2024 it lived mostly in the realm of anecdote and inference; the three cited studies now give it model counts, leakage rates, and inflation magnitudes. Once a problem is stated numerically, its remedies get judged numerically too, which is what places this pilot's "under 5%" overhead figure in an engineering conversation rather than a safety-rhetoric one.

Blinding both sides takes seven steps

The core device in the Double-Blind Evaluation framework is a hardware enclave, and chips such as Intel TDX or AMD SEV-SNP are the parts that make it work. Each derives a private key from secrets burned in at manufacture, and that key becomes the root of trust. Each time an enclave boots it generates a separate ephemeral memory-encryption key and uses it to encrypt everything held in RAM automatically. A cloud provider that physically inspected the RAM or disk recovers no plaintext, and the enclave signs a hash of the software it is running so outside parties can verify what will handle their data.

The report pins the trusted computing base down with three mechanisms. Measurement hashes each TCB layer as it loads and folds the results into an attestation report signed by a key rooted in the CPU vendor's hardware root of trust. Reproducible builds compile every layer deterministically from public source, so any third party rebuilding it obtains a bit-identical artifact and therefore the same measurement. Verification requires each participant, before releasing data, to check the vendor signature chain, confirm the report is bound to a user-supplied nonce, and compare reported measurements against expected ones. No central verification service and no counterparty's verdict has to be trusted.

All of that protects users from the cloud provider; mutual secrecy between evaluator and model owner needs one more layer. Both sides must submit code, and both inference code and evaluation code carry trade secrets. The report's route around this rests on an observation: decompose code far enough and the units of computation stop containing proprietary IP. Code written entirely in open-source frameworks such as JAX or PyTorch is a sequence of method calls whose network behavior is publicly inspectable, so a party can redact its logic and still prove the redacted portion cannot exfiltrate anything. OpenMined's PySyft brokers code submission and approval, and syft-restrict verifies that redacted code calls only allowlisted, non-network-bound methods.

The live procedure runs seven steps: the two parties confirm intent, one launches and pays for the enclave, both request and check the signed software hash, either side optionally submits a mock model or benchmark so the other can build against it, each uploads its possibly redacted code and private assets, each reviews and approves the counterparty's redacted code, and execution releases results only after both approvals land. The party that launched the enclave gains exactly one extra privilege: it can shut the enclave down.

The number this pilot never published is Gemini's score

Nowhere does the technical report state how Gemini 2.5 Flash Lite scored on AILuminate. The experimental setup is otherwise detailed. Prompts were drawn from the AILuminate reserve set, meaning prompts never processed by any model, covering chemical, biological, radiological, nuclear and explosive (CBRNE) hazards, cyberattacks, hate speech, self-harm, and violent crime elicitation. AVERI encrypted and decrypted the prompts and outputs, and AVERI staff evaluated the outputs. There is simply no results table.

That absence describes what was actually shipped. The claim on offer is not about Gemini's safety but about how evaluation gets performed. The report itself says the framework's success depends on whether it inspires enough confidence across separate organizations for them to entrust proprietary assets to it. Because no score was published, the pilot resists being consumed as marketing and instead persists as plumbing available to the next evaluation.

The same fact sets a limit on how much a reader should conclude. Double-blindness guarantees that test items did not leak; it says nothing about whether the model is safe. An uncontaminated evaluation built on a weak benchmark still yields a weak result. AILuminate's hazard taxonomy and violation criteria were produced inside MLCommons AI Risk and Reliability (AIRR) working groups with AVERI representatives participating, and the validity of those criteria is a separate question that cryptography does not answer. Worth noting: the report labels the AILuminate corpus AIRR 1.0 in its abstract and AIRR 1.4 in its experimental setup section.

The bottleneck was legal review, not hardware

The performance conclusion is compact. Running the PySyft Datasite architecture across a two-tier hardware enclave hierarchy on GCP Confidential Space carried under 5% compute overhead. On that basis the report states plainly that the primary bottleneck to secure double-blind evaluation is no longer hardware compute overhead but the procedural overhead and human coordination required for legal agreements and code review.

A technical cost under 5% paired with slow adoption reframes the field's evaluation debate. Explanations that attributed limited external evaluation to performance constraints lose their footing against that number. What remains in the way is contract negotiation and mutual code review, which is a matter of budget, staffing, and standardization rather than silicon.

The stated long-term goal matches the diagnosis precisely: standardized cryptographic attestation pipelines that abstract away dependency hashes and keys with near-zero human overhead, an analogy the authors draw to the HTTPS lock icon in a browser address bar. The point of the lock icon is not that cryptography became easy but that its state became readable instantly by people who know none of it. Double-blind evaluation today sits at the opposite end, where engineers and lawyers on both sides spend weeks to make one run happen.

Three holes the report writes down about itself

The most operationally useful passage is where the authors document their own limits. First, running Gemini 2.5 Flash Lite using only layers present in open-source libraries was judged too large an engineering challenge for this project, so some proprietary method implementations were neither inspected nor allowlisted. AVERI was informed of this and accepted the overall setup. The allowlist verification described above, in other words, was not fully applied in this run.

Second, while the Confidential Space guest OS publishes reference values, opens its source, and has an externally validated build pipeline, individual builds are not independently reproducible because they take private signing keys as inputs. Reproducible builds are the device that makes a measurement meaningful, and at this layer that link is broken. Third, the attestation report is signed and verified through Google services, which keeps Google inside the verification path and raises the amount of trust placed in it.

Taken together, the three locate the pilot precisely. This is not a finished trustless system but an intermediate stage in which the party that must be trusted moved from the contractual counterparty to hardware vendors and a cloud operator. The report says as much: the enclave's guarantees rest on the assumption that the compute provider and hardware manufacturer are unlikely to collude. Trust did not disappear; it relocated to a place with weaker incentives to collude, and that is the design's argument. Organizations will weigh that argument differently, and a government evaluator in particular has reason to examine a structure whose root of trust is a US cloud and a US silicon vendor.

What institutions outside the US should read here

The Singapore AI Safety Institute's role is the most transferable part of the pilot. Singapore AISI evaluated Gemini 2.5 Flash Lite using a private prompt set focused on harmful content elicitation in Singapore's context. Locale-specific safety prompts are expensive to build and lose most of their value the moment they are exposed, and this structure lets such an asset be applied to a frontier model without handing it to the model owner.

Korean-language safety evaluation is exactly that kind of asset. Domestic hate speech, Korean-specific evasion phrasing, and risk scenarios entangled with local law are not substitutable by English benchmarks. The standing dilemma has been that building such a prompt set and then pushing it in plaintext through a foreign vendor's API leaves no defense against the next training cycle. A standardized double-blind path lets external verification proceed without publishing the items, which lengthens the useful life of what a national evaluator builds.

Transferring this pilot's operating conditions directly is another matter. The model under test was Flash Lite, a small member of the Gemini family, running on a single H100. The report states that frontier models are growing beyond trillions of parameters, that the next milestone is extending to many-node H100 and B200 clusters, and that deferred execution frameworks masking internal model architecture will be needed alongside. What was demonstrated is that the procedure holds for a small model, not that large models can be measured this way today. Anyone opening a policy conversation that would require this method should state the outstanding scalability condition alongside it.

Source: Google DeepMind blog announcement of the double-blind evaluation pilot (August 27, 2026) and the technical report "Double Blind Evals: Resolving the Dual Confidentiality Dilemma in AI Safety Auditing" (AVERI, Google, Singapore AISI, OpenMined, MLCommons), compiled by ASAP.