Devin and Perplexity Both Point to Evidence, Not Code, as What Changed With GPT-6 Astra

OpenAI published two customer stories on September 11 and September 14, 2026 in which Cognition and Perplexity name the same core improvement in GPT-6 Astra, and it is not generation quality but the model's ability to test its own work and return evidence that the work runs. Cognition co-founder Walden Yan says Devin returns a recording of the iPhone game Otter Run running in a simulator alongside a report identifying checks that passed and areas left untested, while Perplexity co-founder and Chief Strategy Officer Johnny Ho says his team trusts the model with full end-to-end systems and checks in on it much less frequently than with previous generations. ASAP works only from the statements contained in the two announcements OpenAI published.

What Cognition says it gained from Astra is a recording and a list of what went untested

Cognition is the company behind Devin, an autonomous software engineer used by businesses from big banks to tech-native startups. The problem the announcement names is review rather than generation: as Cognition's engineering teams write more code, reviewing that work has become a challenge.

Walden Yan is explicit about the change. One of the big pieces Astra improves on, he says, is its ability to test and prove that its work actually functions the way you expect. Cognition is applying GPT-6 Astra across its product lineup, including the core cloud agent that is Devin as well as its CLI and desktop products.

The concrete example given is Otter Run. Devin uses Astra to test the iPhone game and returns a recording of the game running in a simulator, alongside a report identifying checks that passed and areas left untested. The recording shows the application's behavior while the report documents the scope of the testing, and engineers use both outputs to inspect how the software works and understand what still needs attention.

The same pattern applies to customer response times. When a customer sends a screenshot of a bug, the team passes it to Devin using Astra, which fixes the issue and returns a screenshot showing the result, according to Yan.

Perplexity has the model build fake external services so the workflow runs end to end

Perplexity trusts Astra with three kinds of work: crafting communications, editing real-world systems, and monitoring production software. Johnny Ho says the company can have the model do all three in a way that previous generations of models were not able to.

The testing approach differs from Cognition's. With limited time to test manually, Ho asks GPT-6 Astra to build a small testing program around an application. The model generates realistic responses like those another service would send, for example a language model API or a connector. By standing in for those services, the model checks how the application responds and tests the workflow from start to finish.

The quoted conclusion is about the unit of trust rather than accuracy. Perplexity is able to trust the model with full end-to-end systems and check in on it much less frequently than with previous generations, Ho says. That is why the announcement leads with trust rather than with a benchmark.

Where the two stories overlap is not testing but the format of the evidence

What follows is ASAP's reading. Cognition and Perplexity differ in industry and product, yet they pulled the same thing out of Astra. Neither asked the model to write better code; both asked it to produce, in a form a human can check, proof that what it wrote runs.

The notable part is that the form is not text. What comes back in the Cognition case is a simulator recording and a screenshot of the fixed result; what gets built in the Perplexity case is an executable test harness that impersonates external services. These are artifacts that can be replayed or run rather than explanations, and they can be judged without reading the code.

That distinction matters because of where the bottleneck sits. When coding agents raise output, the bottleneck moves from writing to reviewing. Cognition's announcement names exactly that problem, and the fact that the company building Devin cites its own code review as a burden shows how general the bottleneck is. Shipping evidence alongside the change does not remove review; it swaps the object of review from code to artifacts.

Changing the object of review introduces its own failure modes

Moving to evidence-based review does not make verification automatically safe, and the goal statement in Cognition's September 2026 announcement is itself written conditionally. Yan says the team expects that over time it has to manually look at less code and ends up shipping more, and the claim that having Devin test its work and provide evidence could help engineers evaluate changes with less manual examination is also framed as a possibility. That is a direction, not a measured outcome.

A structural issue remains as well. When the same actor writes both the code and the tests, verification edges toward self-grading. In the Perplexity case the model even generates the responses of the external services, and whether those responses faithfully represent reality requires separate judgment. Cognition's report format, which names areas left untested alongside the checks that passed, reads as a partial acknowledgment of that risk.

The third issue is the resolution of the evidence. A recording shows behavior that appears on screen, but it cannot capture defects that never surface visually, such as state contamination, race conditions, or permission boundaries. The moment a reviewer feels a video is sufficient is precisely when those defects pass through.

What engineering teams can adjust today is the output specification

The transferable part of these two stories is not model choice but the specification of the request. If the completion condition for an agent task is a code change, review stays entirely with the human; if the completion condition includes running evidence and an explicit list of what went unverified, a meaningful share of review cost moves to the agent.

The second adjustment is demanding the unverified list as its own field. Cognition's report combining passed checks and untested areas looks minor, but forcing an agent to write down what it did not check is a cheap guard against the blind spot of self-grading. That requirement is independent of model generation and can go into the prompts and templates of whatever tool a team uses today.

The third is where the evidence is stored. Once recordings and screenshots become deliverables, the review trail moves from text logs into media, and storage location, retention period, and access control have to be decided alongside it. A recording that contains production screens can itself be sensitive, so the policy for handling evidence belongs in the same document that requires it.

Neither announcement contains a single quantitative metric

The evidentiary weight of these two September 2026 items is limited in three specific ways that are worth stating plainly. First, both are customer stories published by OpenAI on its own site, in interview form. Both the Cognition and Perplexity entries are labeled company size startup, region North America, industry technology, and products API.

Second, neither piece contains quantitative metrics. There is no test pass rate, no reduction in review time, no change in deployment frequency, and comparisons with previous model generations exist only as narrative inside the quotes. Checking in much less frequently, and expecting to look at less code, is the whole of the evidence.

Third, there is no third-party verification. Both companies describe their own experience rather than a controlled comparison against other models under matched conditions. These two items are therefore more accurately read as a signal of what early adopting teams value than as performance evidence for Astra.

The through line is not how much better Astra got, but that the first thing two teams running a frontier model in production asked for was self-verification output. Once generation quality clears a threshold, the remaining bottleneck is review, and the first attempt to break it is a change in the format of evidence. That observation holds regardless of which model name is attached to it.

Sources: OpenAI, "Cognition helps Devin test its own work with GPT-6 Astra" (September 11, 2026) and "Perplexity trusts GPT-6 Astra with end-to-end systems" (September 14, 2026)