OpenAI's Performance Factory Turns Codex Into an On-Call Team
Codex agents investigated more than 1,700 ChatGPT slowdowns in about three months. The lesson for the rest of us is in the config file, the disprove step and one honest 70% number.
Every team that runs a busy service has the same graveyard: a dashboard full of small performance regressions nobody had time to chase. A page got 80 milliseconds slower after Tuesday's deploy. A background job doubled its memory. None of it is an outage, so none of it gets a person.
That backlog is exactly where coding agents were supposed to shine, and for most teams they have not. The agent finds a plausible cause, writes a confident summary, and somebody has to spend an hour checking whether it is true. OpenAI just published how it got past that inside ChatGPT, and the answer is not a smarter agent. It is a stricter workflow.
The post, published November 19, is called "How OpenAI's Performance Factory keeps ChatGPT fast." The headline number is big. In just over three months, the system's Regression Workflow "tracked more than 1,700 investigations and root caused over 600 cases," which OpenAI puts at 17 automated investigations a day.
("Root caused" is engineer shorthand for "found the actual cause.") The more useful part is how unglamorous the design is, and the post includes the one number most agent launches leave out: how often the agents were wrong.
This article walks through the design in plain terms and then turns it into a version you can build this week with the coding agent you already use.
What the Performance Factory actually is
Strip away the name and it is four parts, all described in OpenAI's post:
- A task service that holds work items and the workflow each one follows.
- A dispatch loop that checks which tasks are ready and starts a Codex turn with the instructions for that task's current step. (Codex is OpenAI's coding agent; a "turn" is one round of it working.)
- An API that runs Codex in the cloud, so the agents are not tied to anyone's laptop.
- Subscriptions that wake an agent when something happens, such as a GitHub review comment or a finished deployment.
Think of it like a kitchen ticket rail. Orders hang on the rail, each with a stage written on it. A cook grabs the next ready ticket, does that stage, and hangs it back up. The cook never decides on their own to skip from "prep" to "served."
The agents also run under a service identity, a dedicated account just for them. OpenAI says this lets it "scope its permissions and see which actions came from our performance agents." Every action has a name on it, and the name is not a person's.
The human steps live in a config file
This is the part worth stealing first.
OpenAI's engineers define each workflow with a configuration file and a handful of prompts written in Markdown. In the post's words, "The configuration lists the states a task can move through, the allowed transitions, and which steps need an engineer." The example layout is tiny:
code-change/
├── config.toml
└── work/
├── implementing.md
└── reviewing.md
A state is just a named stage, like "investigating" or "fix proposed." A transition is an allowed move between stages. The point is that the decision about where a person must step in is not left to the agent's judgment in the moment. It is written down, in a file someone can review, before any agent runs.
In the regression workflow diagram, detection is labeled as the system's job, finding the root cause, proposing a fix and checking that the metric recovered are labeled as agent jobs, and "Review & deploy" is labeled as the engineer's job.
That one design choice answers the question most teams argue about forever: how much should the agent be allowed to do? OpenAI's answer is "exactly what the config says, and the config says a human ships."
A second agent whose only job is to say "prove it"
The second idea is the disprove step.
OpenAI found that "agents sometimes stopped at the first plausible explanation they found, and some of those diagnoses turned out to be wrong." Anyone who has used a coding agent recognizes this. It finds a suspicious deploy that lines up with the slowdown and declares victory.
So every proposed diagnosis goes to "a separate Codex session that tries to disprove the proposed diagnosis." It asks the questions a skeptical senior engineer would ask: did similar requests that were not exposed to the change stay healthy? If the suspected change was rolled back, did the metric recover?
The post gives a concrete case. An agent traced failing ChatGPT conversations to errors in image downloads and marked the cause as found. The reviewer noticed that the service that locates files had already recovered while image requests were still failing. The story did not hold together, so the reviewer reopened the investigation. OpenAI runs a second adversarial review on each proposed fix too.
Here is the insight in plain words: the cheapest reliability upgrade for an agent is another agent with the opposite job. One session is rewarded for finding an answer. The other is rewarded for breaking it. You do not need a better model for that. You need a second prompt.
The number that keeps the human in the loop
Now the honest part, and it is why the design above exists.
OpenAI reviewed 100 of the system's root-cause suggestions by hand. "We graded 70% as accurate, with 30% needing further refinement or not fully establishing causality."
Seventy percent is good. It also means three in ten diagnoses needed more work or did not fully pin down the cause. That measures diagnosis quality only, not fixes, and it comes from a team that had built more than 30 internal performance tools before Codex arrived and that backtests its agents against past incidents.
Fixes were harder still. The post says "getting fixes right has been harder than finding the cause," and that agent-proposed changes are sometimes "more complicated than necessary or introduce new issues." Its own example is a good one: adding a cache might speed a page up but serve stale data. That is a product decision, not a debugging step.
So an engineer reviews each fix PR and decides whether to merge it. OpenAI says it is exploring letting agents ship low-risk fixes with no human in the loop, but that is a plan, not the current system.
I find this more convincing than any benchmark chart. A company with every reason to brag about its agents published its miss rate and kept its people at the merge button.
How they test the agents without fooling themselves
One more piece, because it is the part most homegrown setups skip.
OpenAI backtests the agents against past production incidents. Each case starts with a fresh Codex investigator and the original alert, stripped of details that would give away the answer. The investigator "cannot use saved memories (notes from earlier investigations), read the incident writeup, or see results from previous runs." Timing and blame get scored separately, because "an agent can get the timing right and still blame the wrong deployment."
A separate grader also reads the agent's handoff summary and checks whether an engineer who does not know the system could understand what happened and why it matters. And once a day, another agent reads recent runs looking for tool failures and access errors, and opens a task when the same error blocks several investigations.
Put simply: they test the agents on old cases with the answer key hidden, they grade the write-up as well as the verdict, and they have an agent watching the agents' tools break.
Put this into practice: a small version for your team
You do not need a task service or a cloud fleet to copy the useful parts. Start with one recurring kind of work, like "why did this endpoint get slower," and build it by hand.
1. Write the workflow down before you automate it. Create a folder with a short file that lists the stages and marks the human ones. Plain text is fine:
# perf-triage/workflow.md
states: detected -> investigating -> disprove -> fix-proposed -> fix-review -> human-review -> merged
human steps: human-review (always), merged (always)
agent steps: investigating, disprove, fix-proposed, fix-review
This file is the contract. If someone wants the agent to merge, they have to change this file in a pull request where everyone can see it.
2. Write two prompts, not one. An investigate.md that asks for a root cause with evidence (when the slowdown began, which requests are affected, which deploy or config change lines up). And a disprove.md that receives the investigator's answer and tries to knock it down using OpenAI's two questions: did unaffected requests stay healthy, and would rolling back the change actually fix the metric?
3. Run them as separate sessions. With Codex, the noninteractive codex exec command runs a prompt without the chat interface, and version 0.171.0 added codex exec --remote to run sessions on a separate app server (release notes). The plain shell version is enough to start:
codex exec "$(cat perf-triage/investigate.md) Alert: $ALERT_TEXT" > diagnosis.md
codex exec "$(cat perf-triage/disprove.md) $(cat diagnosis.md)" > review.md
The same pattern works with Claude Code's print mode, claude -p. What matters is that the disprove session starts fresh and does not share the investigator's conversation.
4. Give the agent its own account. Whatever credentials the agent uses to read logs and open PRs should belong to a service account, not to you. When something odd happens, you want the audit log to say "perf-agent," and you want to be able to revoke it without locking yourself out.
5. Grade it on old incidents before you trust it. Pick 10 to 20 past regressions where you know the real cause. Feed the agent only the original alert. Score it yourself: right timing, right cause, useful write-up. For a sense of scale, OpenAI hand-graded 100 root-cause suggestions and rated 70% accurate. If your number is lower, you have learned something cheap.
Expect the first runs to be noisy. Expect the disprove step to reject a lot of answers early on. That is the system working.
Honest limitations
This is a write-up of an internal OpenAI system, not a product you can install. There is no download, no price and no public config schema, so the folder layout above is the only real format detail, and the rest is your own build.
The numbers are OpenAI's own. Nobody outside the company graded the 100 suggestions, the post gives no sampling method, and it reports over 600 root-caused cases among more than 1,700 tracked investigations without saying what happened to the rest.
OpenAI's agents sit on top of years of internal tooling: metrics, logs, traces, CPU profiles, memory snapshots and those 30-plus performance tools. An agent with no good telemetry has nothing to reason over, and a disprove step cannot fix missing data.
And the hard part, choosing a fix, still needs judgment the post says agents do not reliably have. If your team expects this pattern to remove engineers from performance work, the source material says otherwise.
Where this leaves you
The interesting thing about the Performance Factory is not that Codex can chase regressions. It is that the safest version of an autonomous agent looks a lot like a well-run team: written procedures, a skeptic in the room, a named account for every action, and a person who signs off.
You can copy all of that without OpenAI's budget. Start with the workflow file and the second prompt. Then measure your own miss rate before you decide how much rope the agent gets.
Sources: OpenAI, "How OpenAI's Performance Factory keeps ChatGPT fast" (November 19, 2026); Codex CLI 0.171.0 release notes (November 20, 2026).