OpenAI's audit of SWE-Bench Pro lands on a number that should freeze anyone quoting coding benchmark scores: 249 of the 731-task public split, 34.1%, are broken. The automated pipeline caught 200 (27.4%). Humans found more.
Chart data
| Label | Value |
|---|---|
| Automated pipeline (27.4%) | 200 |
| Human annotation (34.1%) | 249 |
| Total tasks | 731 |
I counted four issue categories, and they read like a taxonomy of how benchmarks rot. Overly strict tests enforce implementation details the prompt never specified, invalidating functionally correct submissions. Underspecified prompts omit requirements that hidden tests enforce and no model could reasonably infer. Low-coverage tests under-check the requested feature, letting incomplete fixes pass. Misleading prompts actively steer models toward the wrong behavior.
I noticed the gap between review paths. The automated pipeline, combining an initial filter with Codex-based investigator agents, flagged 286 tasks for five engineers to independently annotate. Their judgments overlapped with the agent pipeline in 74% of cases, but humans called more tasks broken. The agent saw ambiguity where the engineer saw a defect.
The timing matters. Frontier models reportedly climbed from 23.3% to 80.3% pass rate on SWE-Bench Pro in eight months. OpenAI now advises developers to "carefully examine" those results. A benchmark where a third of tasks are broken cannot validate a 57-point jump.
Some of that gain is real model improvement. Some is models learning to satisfy broken tests. This follows a prior investigation into SWE-bench Verified, which had its own design and contamination issues.
Or defensively expect models to be stupid. Seems the smart thing to do is not assume an agent will do the right thing. But to create the scaffold / harness that enforces constraints to steer them towards a good result. Then you can swap out the really smart model for maybe something cheaper.
The Hacker News thread tracks the finding. The sentiment is consistent: trust the scaffold, not the score. Build private benchmarks. Expect the model to fail and design around it.
This echoes what I noted about GeneBench-Pro, which shipped with 129 synthetic problems and zero model scores, and the SWE-agent bug-fixer whose creators told users not to deploy it. The pattern is consistent: coding benchmarks and coding agents both struggle to prove their own reliability.
OpenAI deserves credit for auditing its own field's scoreboard. The question is whether the next benchmark will get the same treatment before the numbers land on a marketing slide.