Hot AI News

OpenAI’s agents now investigate 17 ChatGPT slowdowns a day, and engineers just review

OpenAI says its internal Performance Factory tracked more than 1,700 regression investigations in about three months, with Codex agents diagnosing slowdowns and engineers approving the fixes.

OpenAI’s agents now investigate 17 ChatGPT slowdowns a day, and engineers just review
Image: OpenAI

OpenAI has described an internal system called the Performance Factory, in which Codex agents investigate ChatGPT slowdowns and propose fixes. In an engineering post, the company says it tracked more than 1,700 investigations in just over three months and root caused over 600.

OpenAI puts that at about 17 automated investigations a day. The system is internal, and the post describes no customer product, price or release date.

Performance Factory orchestration workflow: alerts, engineers and other agents create tasks that agents pick up
Image: OpenAI

From a firing alert to a verified recovery

The first workflow built on the Factory is the Regression Workflow. A configured performance alert creates a task. One agent works out why things got slower, writes what it found into that task and drafts a fix for an engineer to review. After deployment, a different agent takes over the same task and checks whether performance is back to baseline.

OpenAI lists four components behind this: a task service where engineers define workflows in Markdown and configuration files, a dispatch loop that assigns ready tasks to agents, an internal API that runs Codex turns in the cloud, and subscriptions that wake an agent when something changes, such as a GitHub review comment or a finished deployment. According to OpenAI, agents run under a service identity with access to repositories and observability data such as metrics, logs, traces, CPU profiles and memory snapshots, so their permissions can be scoped and their actions traced.

The evolution of performance engineering from manual work to reusable skills and agent factories
Image: OpenAI

The diagnosis relies on a shared skill called perf-forensics, which tells Codex which tools exist and how to work through a slowdown: when it began, which requests were affected, and whether a deployment or configuration change fits. If a fix only partly restores performance, the agent closes the original task and opens a new one for the remainder.

Advertisements

A second agent tries to prove the first one wrong

OpenAI is unusually open about error rates. Early agents sometimes accepted the first plausible explanation, so the team added an adversarial review, in which a separate Codex session tries to knock down the diagnosis. In one case, an agent blamed image download errors for failing conversations. The reviewer noted that the file-locating service had already recovered while image requests still failed, and reopened the case.

OpenAI's agents now investigate 17 ChatGPT slowdowns a day, and engineers just review
Image: OpenAI

In a manual review of 100 root cause suggestions, OpenAI graded 70% as accurate. The other 30% needed more work or did not fully establish causality. Fixes proved harder than diagnoses: the post says agent-proposed changes were sometimes more complicated than needed or introduced new problems, so proposed fixes now get the same adversarial check. Architectural trade-offs, such as a cache that speeds things up but serves stale data, still go to an engineer.

These figures are OpenAI’s own and come from its own review. The post does not say how many of the roughly 600 root-caused cases led to a merged fix, or how much latency the system has recovered.

A track record of turning Codex on OpenAI’s own stack

The post fits a pattern of OpenAI describing its infrastructure through the tools it sells. On the same page it points to recent engineering write-ups on the Codex sandbox, on scaling Git and on online storage for more than 1 billion ChatGPT users, all published in the past month and a half.

The performance work matters because the product keeps widening. Our earlier coverage shows ChatGPT for Sales connecting 30+ tools in one chat, and GPT-6.2 Astra listed at $8 per million tokens, $2 below GPT-6 Astra. Every added integration or cheaper model drives more traffic and more releases, and each release is a chance for a regression. OpenAI frames the Factory as the answer to that pace, saying traditional performance engineering became unsustainable given how fast agentic development moves.

The six-stage path OpenAI describes is also a practical template for other teams: manual investigation, Codex as a partner, reusable skills, a human copying alerts into Codex, automated cloud investigations, then a stateful factory. The company’s own lessons are modest. It says the biggest risk is flooding engineers with findings, so related incidents are now linked, grouped by the metrics teams care about and ranked by estimated impact. It tests changes to skills and models by replaying past production incidents with the answer stripped out, and runs a daily agent to catch broken tools and access errors in earlier runs.

Advertisements

What OpenAI still has to prove

The remaining bottleneck, as OpenAI describes it, is the wait for a human to approve and merge each fix. The company is exploring agents that ship low-risk changes without a human, along with synthetic performance tests and proactive scans of profiles and traces to spot slowdowns that no alert catches. None of that is described as running today.

OpenAI closes with a line that doubles as its pitch: “Codex already helps us build ChatGPT faster. Now it’s helping us keep ChatGPT fast.” Whether outside teams can reproduce the results is open. The post publishes no code or external access, and the numbers rest on one company’s own services and tooling.

Sign Up for the Daily AI Pulse

One email a day. All the stories that matter.

Alex CarterAI News & Big Tech Correspondent

Alex Carter writes for Aiholics, keeping readers updated on the fast-paced world of AI and Big Tech.

Leave a Reply

Your email address will not be published. Required fields are marked *

You may also like