OpenAI cut ChatGPT’s read latency 12x, from 120 ms to 10 ms

OpenAI has published Part II of its engineering write-up on Habitat, the storage platform behind ChatGPT, describing how it tuned Azure Cosmos DB. According to OpenAI, global read p50 latency fell from roughly 120 ms to roughly 10 ms this year.

Shared containers, then dedicated ones, then shared again

Habitat serves more than 1 billion users across 40 geographic regions and holds over 500 petabytes, OpenAI says. Its largest Azure Cosmos DB account stores tens of petabytes and handles more than 20 million queries per second. The company believes it is the only one running Cosmos DB at that scale.

The post describes three phases. At first, all products shared one default container. Clients picked their own partition keys, so common keys such as user_id collided across products. Limits that Cosmos DB enforces at the physical partition level let niche features starve core login and signup flows. When partitions split as data grew, each child inherited a smaller share of the container’s throughput, so storage-heavy workloads lost capacity while traffic stayed flat.

OpenAI then moved critical new use cases to dedicated containers. That contained outages to single features, but it created a planning burden. A feature going from load-test traffic to a million requests per second could be 10,000% of an empty container’s headroom, and teams that forgot to pre-scale ended up in incident rooms on launch day.

The third phase returned to one shared container, now with fixes built by the Azure team: throttling of the hottest keys first, priority tiers so bulk jobs are throttled before interactive requests, and a reworked serverless mode. In that mode each physical partition can scale to 10K RU/s and split on request-unit usage as well as storage. OpenAI reports no provisioning or noisy-neighbor incidents in the months since.

Two outages that forced a joint operating model

OpenAI also recounts two incidents from late 2024. A power outage hit the primary Azure region of one of its largest accounts, and the failover turned out to take several hours, a scale nobody had tested. About a month later, failing back triggered a connection stampede that overloaded the original region’s gateway, because the gateway-mode client sent all traffic there even though only some partitions had moved.

The response was closer integration. The two teams now monitor every layer together, test against a production-sized non-production account, iterate on SDKs jointly and can page each other directly. Changes include gateway load protection and metadata caching, connection pooling, and separate VPCs for critical and offline workloads.

Caching, Rust and change feeds drive the latency gain

Reads make up over 90% of Habitat requests. OpenAI credits three changes for the 12x gain: replicating small, latency-critical datasets such as authentication metadata into nearby regions (some accounts span all 40 regions), an in-region Valkey cache, and a rewrite of the Habitat service from Python to Rust.

Cache freshness relies on Cosmos DB’s change feed, which propagates invalidations globally within seconds. OpenAI compares the design to Meta’s TAO. The same change-capture pipeline, with millions of changes per second flowing through more than a hundred Kafka topics, also feeds secondary indexes and analytics.

The slowest requests still go to distant storage, the company admits. Its longer-term answer is an in-house, exabyte-scale store that combines Rockset’s RocksDB Cloud core with Raft replication and the same document model as Cosmos DB. It trades lower peak query rates for less flash capacity, and it is still in development.

What the numbers mean outside OpenAI

This is an engineering account, not a product launch, and nothing here is available to customers. The figures are OpenAI’s own and no outside party has verified them. The post gives no cost data, no tail latencies and no date for the in-house system.

Microsoft’s public documentation shows how far OpenAI is from typical use. It lists a limit of 10,000 RU/s per partition and 20 GB of storage per logical partition. Containers top out at 1,000,000 RU/s unless a support ticket raises the cap. The per-key ceiling explains why hot keys were such a problem, and the container cap shows why an account serving 20 million queries per second is an outlier that needs special engineering with the vendor.

The strategic signal is a plan to move off a managed database for the heaviest data. OpenAI says the in-house system is aimed at the difficulty of replicating data for the next billion users, and that it is tuned for bulky data types. Meta’s TAO, which OpenAI cites as a model, was described in a 2013 USENIX paper as handling about a billion reads per second, so the pattern of a custom cache-heavy layer over a general store is well established.

Signs to watch are whether the shared-container design keeps its incident-free record as launches accelerate, whether Azure ships the new serverless mode and throttling features to other customers, and whether OpenAI gives a timeline for the exabyte-scale store. Part I of the series covered Habitat’s move from a Python client to a service, and the Rust rewrite described here suggests that approach has reached its limits.