Codex Agents Ran 1700 Investigations Into ChatGPT Slowdowns and Found the Cause in 600
When agents write the code, ChatGPT can slow down after any rollout, and engineers can't track down the cause by hand fast enough. At OpenAI, an alert about a speed drop now launches a Codex agent that looks for the cause and proposes a fix, which an engineer then reviews. In a little over three months, the system ran more than 1700 investigations and found the cause in 600+ cases, about 17 a day. Teams that ship often should hand root-cause hunting to agents and leave the decision to deploy to humans.
AI-processed from OpenAI Blog; edited by Hamidun News
Imagine that after a routine rollout, ChatGPT starts responding slightly slower. For example, the so-called time to first token, the pause between a question and the start of the answer, goes up. In the past, an OpenAI engineer would dig through metrics, logs and traces and work out by hand which change was to blame.
Now agents write the code, releases keep multiplying, and investigating every slowdown manually is no longer feasible. So the company built Performance Factory, a cloud system based on Codex: when a slowdown alert fires, an agent investigates the cause on its own and proposes a fix, and a human reviews it. In a little over three months, the system's first working scenario tracked more than 1700 investigations and established the root cause (that is, the specific change that caused the slowdown) in more than 600 cases.
That works out to about 17 automated investigations every day. For your team, this is an example of how to take routine root-cause hunting off people's plates without handing agents the decision about what reaches production.
How OpenAI Got to a "Factory"
The path took six steps. At first, everything was done by hand: engineers looked at metrics, compared traces, debugged live machines and read code. For recurring problems, such as drops in time to first token, the team wrote tools, and over time they accumulated more than 30, including CPU and memory-allocation profilers.
Then Codex became a partner: an engineer asked "why did it get slower?", and the agent found the relevant metrics, traces and code, even in an unfamiliar system. But a human still had to steer the investigation. So the team's process was written down as a shared skill called perf-forensics. It tells the agent which tools are available, when to use them and how to proceed: find the moment the slowdown began, identify the affected requests, and check whether rollouts or configuration changes explain it. The skill was tested on evals built from past regressions, and for selected types of problems (for example, drops in ChatGPT speed and availability), Codex started producing a diagnosis in a single run.
Next, engineers began to feel like a "human proxy": an alert arrived, a person copied it into Codex and pasted the result back into Slack. The investigations were moved to the cloud, and an alert now started a session by itself. But automation stopped there: a human still wrote the fix and checked it after rollout, and figuring out which regressions needed attention at all meant wading through Slack threads. Performance Factory closed the whole chain.
What the System Consists Of
There are four components. The first is a task service: engineers describe workflows with configuration files and Markdown prompts that specify the states a task can be in, which transitions are allowed and at which steps a human is needed. The second is a dispatcher, which periodically checks which tasks are ready for the next step and launches Codex for them with instructions for the current state.
The third is a cloud API, which runs Codex inside OpenAI's network with access to repositories and observability tools: metrics, logs, traces, CPU profiles and memory snapshots. The agent works under a service account, so its permissions can be restricted and its actions are easy to tell apart from human ones. Finally, there are subscriptions: a task that is waiting for a human or a rollout "goes to sleep" and wakes on an event, such as a comment on a pull request or a failed CI check.
The first workflow is the Regression Workflow. An alert opens a task, the agent finds the cause and proposes a fix, an engineer reviews it and decides what to roll out, and after deployment another agent checks whether the metrics have returned to their baseline. If the regression persists, the task goes back to investigation. State and conclusions are stored in the task itself, not in a single agent's memory, so other agents or people can pick up the work.
Arguing With Itself, and the Difficulty of Fixes
The agent looks for when the metric first got worse (sometimes several hours before the alert) and compares groups of requests: did similar requests actually get slower, or did the share of already slow requests simply grow? It then examines rollouts and flag changes, reads the code and uses logs and traces to check whether a change could have produced such an effect.
At first, agents often stopped at the first plausible explanation, and some diagnoses turned out to be wrong. The fix was adversarial review: a separate Codex session tries to refute the diagnosis. If a rollout is to blame, did similar requests that it didn't touch stay healthy? If the change was rolled back, did the metric recover? In one case, an agent linked failures in ChatGPT conversations to image upload errors and considered the cause found. The reviewer noticed that an internal file search service had already recovered while requests with images were still failing. The diagnosis didn't explain the symptom, so the investigation was reopened.
In a manual review of 100 proposed causes, 70% were judged accurate, while 30% needed more work or didn't prove a causal link. So the team split the large task into narrow Codex steps: some look for the cause, others challenge it, and still others propose a fix. The authors call the fix itself the hardest part: after the diagnosis is checked, Codex writes a patch and opens a pull request, and the decision stays with the engineer.
- Nov 19, 2026Anthropic goes public and raises $81.3 billion, beating SpaceX's $75 billion record
- Nov 19, 2026Infostart connects Direct, Metrica and Wordstat to ChatGPT: 6 commands, shared access
- Nov 19, 2026Amazon and Google's Anthropic Stakes Valued at $308 Billion in IPO at $83 a Share
- Nov 19, 2026Google's Gemini 4 Neon runs twice as fast as Carbon and costs 17% less per task
Key Questions Answered
How OpenAI Got to a "Factory"?
The path took six steps. At first, everything was done by hand: engineers looked at metrics, compared traces, debugged live machines and read code. For recurring problems, such as drops in time to first token, the team wrote tools, and over time they accumulated more than 30, including CPU and memory-allocation profilers. Then Codex became a partner: an engineer asked "why did it get slower?", and the agent found the relevant metrics, traces and code, even in an unfamiliar system.
What the System Consists Of?
There are four components. The first is a task service: engineers describe workflows with configuration files and Markdown prompts that specify the states a task can be in, which transitions are allowed and at which steps a human is needed. The second is a dispatcher, which periodically checks which tasks are ready for the next step and launches Codex for them with instructions for the current state.
What are the key details on Arguing With Itself, and the Difficulty of Fixes?
The agent looks for when the metric first got worse (sometimes several hours before the alert) and compares groups of requests: did similar requests actually get slower, or did the share of already slow requests simply grow? It then examines rollouts and flag changes, reads the code and uses logs and traces to check whether a change could have produced such an effect. At first, agents often stopped at the first plausible explanation, and some diagnoses turned out to be wrong.
Build & Ship Production AI Agents Without Boilerplate Code
Start your free 72-hour trial at Hamidun Academy (academy.hamidun.com) — interactive lessons on Claude Code & Cursor, live Tech Radar, and production agent templates.
The AI world, distilled — once a week
Seven stories that actually mattered, hand-picked. No noise, no reposts, no press releases.
Done! Check your inbox for a confirmation.