Oscar Crowe Artificial Intelligence OpenAI’s Performance Factory: Codex Agents That Keep ChatGPT Fast

OpenAI’s Performance Factory: Codex Agents That Keep ChatGPT Fast

OpenAI published an engineering account on November 19, 2026 describing an internal system called the Performance Factory, a cloud setup in which Codex agents investigate ChatGPT slowdowns, propose fixes for engineers to review, and then check that performance actually recovered. In just over three months, the first workflow tracked more than 1,700 investigations and root-caused over 600 cases, an average of 17 automated investigations a day. The post, written by OpenAI engineers Okay Zed and Cyrus Forbes, offers a rare look at how a large AI company uses agents to maintain its own product.

The Problem: Regressions Faster Than People

A performance regression is a change that makes a service slower than it used to be. OpenAI's example is first token latency, the delay before the first piece of a ChatGPT reply appears. Teams set alerts on measurements like that, and an alert fires when a number crosses a threshold. It tells you something got slower. It does not tell you why.

Before Codex, OpenAI's performance engineers investigated by hand, reading metrics and logs, comparing traces, and debugging live machines. Over time they built more than 30 specialized tools, such as CPU and allocation profilers. The work was slow, and knowing where to look required deep knowledge of the service. As agentic development accelerated, the team found that manual performance engineering could not keep up with the pace of its own roadmap.

How the Team Got to a Factory

OpenAI describes six stages. First came manual work. Then Codex became a partner that engineers could ask "why did this get slower?" The team captured its method in a shared skill called perf-forensics, which tells Codex which tools exist and how to work a slowdown: find when it began, identify the affected requests, and check deployments and configuration changes. Once that skill existed, an alert could produce a diagnosis in a single run. Next the team connected alerts to cloud Codex sessions so the work started automatically. The final step was the factory, which tracks what each task needs next and assigns an agent when a step is ready.

The Regression Workflow

The first workflow inside the factory is the Regression Workflow. It opens a task when a performance alert fires. An agent investigates, records its findings in the task, and proposes a fix for an engineer to review. After deployment, another agent picks up the same task and checks whether performance returned to baseline. A regression moves through four stages: needs investigation, root caused, fix proposed, and resolved. The task stores its status and findings outside any single agent's context, so another agent or an engineer can take over at any point.

The factory has four components. A task service stores tasks and progress, with workflows defined in configuration files and Markdown prompts. A dispatch loop checks which tasks are ready and starts a Codex turn with the instructions for the current state. An internal API runs Codex in the cloud inside OpenAI's network, with access to code repositories and observability tools such as metrics, logs, traces, CPU profiles, and memory snapshots. Subscriptions wake an agent when something changes, like a GitHub review comment or a finished deployment. Agents run under a service identity so OpenAI can scope their permissions and see which actions came from the performance agents.

Adversarial Review and the Honest Numbers

OpenAI found that agents sometimes stopped at the first plausible explanation, even when the evidence did not fully support it. The fix was adversarial review: a separate Codex session starts with fresh context and tries to disprove the diagnosis, checking assumptions and testing counterfactuals. The reviewer now sends back about 25 percent of diagnoses. In a manual review of 100 root-cause suggestions, OpenAI graded 70 percent accurate, with 30 percent needing more work or not fully establishing causality.

The gap between diagnosis and repair is where the numbers get humbling. Of the more than 600 investigations that reached a root cause, around 400 progressed to a proposed fix. Roughly 50 shipped fixes were influenced by or matched agent suggestions, and about 15 were pull requests authored directly by Codex. OpenAI says Codex does best when the cause is clear and the fix is contained, like a change that makes a service serialize the same conversation several times. Architectural changes are harder, and a caching fix can introduce stale data. Engineers still review and approve every merge.

What It Means for Engineering Teams

The pattern generalizes. Split a recurring job into explicit states, give agents the steps that suit them, and keep a person on the decisions that carry tradeoffs. Store findings and reproducible queries rather than raw telemetry so work can pass between agents without blowing up context. And monitor the agents themselves. OpenAI runs a daily agent that scans recent agent logs for recurring tool failures and permission problems, then opens a task to fix what keeps blocking investigations.

OpenAI's account is worth reading as a template rather than a mystery. The team did not jump to a fully autonomous factory. It went from manual work, to Codex as a partner, to a reusable skill, to cloud sessions, to a stateful workflow, and only then to the idea of a factory. Each stage removed one human bottleneck and added one control. That sequencing is the part most teams skip. They want the factory before they have the skill, and the skill before they have the evals.

OpenAI says it is working on proactive optimization, where agents compare CPU profiles, memory data, and request traces across releases to find slowdowns that alerts miss. It is also exploring whether agents can deploy low-risk fixes without human approval. The bottleneck today, the company says, is the engineer who reviews and merges the fix.

For more on AI agents, model releases, and the business of AI, visit crowe-endeavors.com.

Sources

"Gemini Computer Use on Mac + OpenAI's Performance Factory (Nov 20)" (Agent Signal (YouTube), November 20, 2026) — https://www.youtube.com/watch?v=VedGa0dqsNg

"How OpenAI's Performance Factory keeps ChatGPT fast" (OpenAI, November 19, 2026) — https://openai.com/index/how-openai-performance-factory-keeps-chatgpt-fast

"Exclusive: OpenAI reveals Performance Factory, a Codex-powered software optimization system that keeps ChatGPT fast" (VentureBeat, November 19, 2026) — https://venturebeat.com/orchestration/exclusive-openai-reveals-performance-factory-a-codex-powered-software-optimization-system-that-keeps-chatgpt-fast

Leave a Reply

Your email address will not be published. Required fields are marked *

Related Post