OpenAI Ran 70 Million Requests Per Second on Python Before Rewriting Habitat in Rust

OpenAI disclosed on September 11, 2026 that Habitat, its online storage platform, now handles more than 70 million requests every second and over 500 petabytes of data, supporting products used by more than 1 billion people each week across almost 40 geographic regions. Habitat began in mid-2024 as a small Python library talking to a single Azure Cosmos DB, and in Q2 2026 two engineers working with Codex and GPT-5.5 rewrote the entire service in Rust. ASAP works only from the figures and design decisions stated in the part-one post by OpenAI members of technical staff Jon Lee, Chaomin Yu, and Ben Ries.

Habitat is the layer built to delete databases from product code

Habitat started from a simple premise: product engineers should not need to think about database management. It began in mid-2024 as a small Python library interacting with ChatGPT's main server, supporting a small set of operations that mapped under the hood to Azure Cosmos DB.

The work the library absorbed is listed explicitly in the post. Schema lookup, routing, authorization, encryption, serialization, request shaping, and connection pooling all moved behind the library. Product engineers did not even need to consider whether data came from Azure Cosmos DB, from caches, or from another type of storage.

Adoption followed. The post notes that Habitat spread rapidly among OpenAI product engineers despite no concerted central push away from self-serve Postgres and Azure Cosmos DB. Adding features such as client-side caching, compression, or encryption to the shared library was easy enough that product developers did it themselves.

The move from library to service was triggered by the outage it was meant to prevent

The transition came by the middle of 2025, and the reason was deployment coordination cost. When the team wanted to reduce the blast radius of a single region outage by migrating its most critical data sets to regionally distributed Azure Cosmos DB accounts, it had to introduce routing logic into the client behind a feature flag, roll that out to every client, and only then enable the flag.

Coordinating deployments across dozens of services took days. Adding shadowing to verify the sharding logic took another couple of days. A bug fix took another couple of days. When the team was finally ready to enable the flag, one team rolled its service back to a previously buggy client for unrelated reasons, causing exactly the outage the migration had been designed to avoid.

Decoupling storage logic into a standalone service gave OpenAI a single point of control for deployments, observability, and platform enhancements. The post also frames it as a security decision: the service is a single chokepoint where access control policies are centrally enforced, audit logging happens, and access to underlying storage resources such as Azure Cosmos DB is limited, protecting user data from external, internal, and agent actors.

Staying on Python was not a performance judgment but a debt schedule

OpenAI states plainly that it chose Python for the service knowing the choice was suboptimal. The post calls it a strategic incursion of technical debt, and says the primary objective was not cost or resource optimization but unblocking product developers and achieving platform stability. The team also recognized that Python's inefficiencies would not be acceptable at 100x scale, making an eventual rewrite almost certain.

What follows is ASAP's reading. This passage is unusual among infrastructure write-ups because it treats technical debt as a schedule set in advance rather than an excuse written afterward. The standard debt narrative runs in the order of shipping something rough under pressure and repaying it later; Habitat decided when and how it would repay before taking the debt on.

The notable part is what the repayment was bet on. The post says the team made a calculated wager that the rapid advancement of its own coding models would simplify the technical path, betting that by the time a full migration off Python was required, Codex and GPT would make it achievable, and that the bet proved correct. Very few organizations can put the future performance of their own product into their infrastructure roadmap as a premise, which makes this particular decision hard for other companies to copy directly.

The culprits behind tail latency were config parsing and connection reuse order

The real problem with the Python service was tail latency rather than throughput. Because an average user request results in hundreds of database calls, the slowest database call is the one the user feels, as the post explains. Asyncio provides I/O concurrency but does not work around the Python GIL to provide CPU parallelism, and Habitat carried plenty of CPU work: routing, compression, encryption, checksumming, downstream health checking, request shadowing, and hedging.

The measurement method is simple. By periodically scheduling background tasks and recording the delta between expected and actual execution time, the team measures event loop scheduling delay empirically in real time. At high utilization, even modest numbers of concurrent requests per process produced scheduling jitter up to hundreds of milliseconds, and several seconds in some edge cases. The response was to keep each process serving only a small number of concurrent requests and massively scale out the number of Python worker processes.

The first culprit was feature flags. Statsig was configured by default to poll for refreshed configs every minute with no jitter, and the config included every production rule across every service. Separately, an architectural decision ran up to 8 Python processes per pod. Combined, every minute each pod had a moment where all of its workers stalled in-flight requests and spent their CPU cycles parsing a giant configuration file. The fix was a smaller targeted config, a longer refresh interval, and jitter on background tasks.

The second culprit was connection reuse order. Python's aiohttp TCPConnector defaults to LIFO reuse, selecting the most recently returned connection for the next request. During a burst, slower overloaded servers returned connections later and were therefore selected more frequently, gradually concentrating traffic on pods already struggling. Before load balancing was adjusted, tail processes served 5 to 10 times the number of concurrent requests as the average. Patching the pool to FIFO reuse broke the feedback loop and reduced steady state request variance as well.

Scaling out processes sends the bill downstream

The cost of running an order of magnitude more Python processes is a thundering herd problem that shows up on downstream dependencies rather than in the service itself. The post describes it concretely: an untuned daily deployment can cause significant CPU churn from connection cycling, and a connection leak can take out the network by saturating the NAT gateway. These are not uncommon problems elsewhere, but the threshold for triggering them drops sharply when a service runs an order of magnitude more processes.

The mitigation layer is Envoy. It upgrades Python's HTTP/1 connections to HTTP/2 to take advantage of multiplexing, then pools those connections and extends their lifetimes. Rate limits and circuit breakers, which would be less effective spread across standalone Python processes, are centralized there too. Today OpenAI mostly depends on Istio and Envoy for connection pooling and load-aware balancing, avoiding the problem altogether.

Deliberately weakening the query API is what carried Python this far

The reason Python scaled as far as it did, according to the post, is Habitat's constrained API. Rather than allowing clients to construct arbitrary SQL queries that could cause large table scans or joins across many tables, Habitat exposes a simple NoSQL API. The lack of a powerful API is described as an explicit tradeoff in the design.

The evidence offered is the Postgres era. Early on it was easy to review every query and schema change to confirm it operated against indexed data, but as the team and products grew this became unmanageable and was a frequent cause of outages where a single expensive new query on a hot path took out the database. The post names the underlying issue as cost imbalance: it is cheap and easy to write SQL queries that are expensive and hard to run.

The data model is an object-and-edge structure inspired by TAO. Clients predefine objects and edges and how they relate, but not the content of each type, and while the result resembles a graph, Habitat does not support typical graph traversal queries outside of querying direct edges of a particular object. Each object and its edges are colocated in a storage-level partition, but no database-level effort is made to colocate objects with the remote objects their edges point to. Horizontal partitioning becomes easy while graph traversals become inefficient, since any hop may require fetching from two entirely different Azure Cosmos DB accounts in different regions.

Teams with complex query needs get a separate escape hatch. Change data capture streams changes from online storage to isolated Rockset instances in near-real-time, and each client team scales its own Rockset instance. The provisioning adds friction, but the post judges it the right tradeoff at this moment: simple queries as the default, with an escape hatch for those who need complex ones.

Two engineers rewriting the service in Rust in one quarter is the conclusion of part one

The rewrite landed in Q2 2026 with 2 engineers, Codex, and GPT-5.5. That combination rewrote the entire service in Rust, and the new Rust service now handles 95% of production requests, with Python to be deprecated entirely in the coming weeks. The figures OpenAI published are that the Rust service is 6x more CPU efficient and 15x more memory efficient than the Python version, with significantly lower average and tail latencies.

Habitat's standing just before the rewrite was disclosed as well. It was the second largest service by core count at OpenAI and fourth by Envoy footprint. At its peak, Python helped serve more than 20 million requests every second.

These numbers deserve to be read in two parts. The impressive half is the headcount: finishing a language migration for a system taking tens of millions of requests per second with two people in a single quarter unsettles the usual way rewrite cost is estimated in staff-months. The half to read carefully is the multiples. A 6x CPU and 15x memory gain sits within the ordinary expected range for Rust over Python, so those figures are not themselves a result produced by coding models. What the models contributed is migration time, not the efficiency multiple, and the two should not be conflated.

What most teams can take from this is the ordering, not the scale

Few organizations will ever serve 70 million requests per second, but the reusable part of this post is the order in which decisions were made. First, measuring scheduling delay as its own metric. OpenAI states that for Python services, monitoring how busy the asyncio loop is and tuning accordingly is critical in addition to standard utilization and saturation metrics on memory, CPU, network, and disk. Any team running a Python or Node API server can add that instrumentation today.

Second, the boundary between a library and a service. A shared client library is fast early on, but the cost structure inverts the moment a single change requires coordinating dozens of services. The trigger OpenAI used to decide was not performance but a deployment coordination failure, and that trigger applies at any scale.

Third, the choice to weaken the API on purpose. Giving a shared internal data layer more capability feels generous, but Habitat's lesson is that allowing only predictable, constant-work requests and pushing complex queries onto a separate read path makes operations simpler. The crucial detail is that the restriction shipped with an escape hatch; without the Rockset path, the constraint would have collapsed under workarounds.

What part one leaves open is the conditions behind the comparison

The gaps in the post are worth stating plainly. The largest one is the conditions behind the Rust figures. Which workloads and traffic mix produced 6x CPU efficiency and 15x memory efficiency, and how much latency actually improved, are not specified; the post says only that more learnings will be shared in a future blog.

The second gap is the division of labor in the rewrite itself. Beyond the composition of 2 engineers, Codex, and GPT-5.5, nothing is said about the proportion of model-generated code, the review process, or whether incidents occurred during the migration. For readers trying to judge the real-world performance of coding models, that is precisely the missing information.

The third gap is the structure of the series. Multi-tenancy reliability at scale, the layered strategy for optimizing read performance, and the scaled partnership with Azure Cosmos DB are all deferred to part II. Since the storage layer that actually absorbs 500 petabytes and 70 million requests per second is the subject of that post, part one alone is too early a basis for evaluating Habitat's scalability.

The through line is sequencing rather than scale. While growing more than 10x year-over-year for three consecutive years, OpenAI chose which problem to block next instead of choosing the optimal implementation, paying Python's inefficiency as interest to buy product velocity. Repaying that debt with its own coding models is the real subject of part one.

Source: OpenAI engineering blog, "Rapidly scaling online storage to serve over 1 billion ChatGPT users" (September 11, 2026, by Jon Lee, Chaomin Yu, and Ben Ries)