Contents
Every few months a new coding benchmark claims a frontier model “writes production code.” Then someone runs that same model against an actual production codebase, one with tax rules, half-documented billing logic, and three ways of doing the same thing because three different engineers touched it over five years, and the score falls off a cliff. That’s exactly what happened this week. Specific Labs, a YC-backed startup, published Real-SWE, a benchmark built entirely from private codebases it licensed from real companies, including a consumer events app with 200K+ users and a fintech platform processing 100K+ bank statements. The best model on it resolves 38.8% of tasks. On SWE-bench Verified, frontier models routinely clear 70-90%. That gap is the story, and it matters to anyone deciding whether to let an agent loose on their own repository rather than a curated GitHub issue.
Why the gap is so wide
SWE-bench and its successors pull tasks from public GitHub repositories, which means a meaningful share of that code, and sometimes the fix itself, sits somewhere in a model’s training data. Real-SWE closes that door on purpose. Every task comes from a private, out-of-distribution codebase, so agents can’t lean on memorized patches or documentation. The tasks are also things engineers actually got assigned, like making sure tax is calculated correctly per business rules and an external tax authority API, or migrating a billing schedule without breaking reconciliation. The median reference fix touches 11 files, against 6 for comparable benchmarks. Real work rarely stays inside one file.
What the numbers actually show
Across eight model-and-harness combinations, ten tasks, and 640 scored rollouts, Fable 5.1 running on Claude Code leads at a 38.8% resolution rate, ahead of GPT-6 Astra on Codex CLI at 33.8% and Gemini 3.8 Flash on Gemini CLI at 31.2%. Six of the ten tasks resolve below 15% for every model tested. One task, a stream reducer for an analytics pipeline, went 0 for 64 across every model and every rollout. Cost tracked loosely with resolution rate at best: Gemini 3.8 Flash was both the cheapest rollout at roughly $2.50 and the third-best performer, while GPT-5.6 Sol cost more than that and finished last.
The more useful data isn’t the leaderboard, it’s the failure taxonomy Specific Labs applied to every failed rollout: unverified assumptions, missed requirements, integration errors, regressions, and fixes landed in the wrong file entirely. The mix differs sharply by model. Grok 4.6 fails almost entirely by missing requirements outright, 67% of its failures. Gemini 3.8 Flash rarely misses a requirement but wires its fix into the surrounding system incorrectly 49% of the time. GPT-5.6 Sol’s most common failure is building on an assumption it never checked against the actual code. That’s a genuinely different diagnosis for each model, and it argues against picking a coding agent off a single aggregate score.
Try it: explore the failure breakdown
Switch between models and watch the dominant failure mode change shape entirely rather than just the percentage. That’s the part a single leaderboard number hides.
What I’d take from this
If you’re evaluating a coding agent for your own codebase rather than a demo repo, the resolution rate on any public benchmark is close to irrelevant. What matters is which failure mode your codebase is most exposed to. A codebase full of implicit business rules and undocumented conventions will punish unverified assumptions hardest, which is where GPT-5.6 Sol struggled most. A codebase with many integration points and shared services will punish integration errors, where Gemini 3.8 Flash struggled most. Specific Labs says it plans to keep the underlying task set private and rotate it, precisely so agents can’t be tuned against it the way public benchmarks eventually get gamed. That’s the right instinct, and I’d like to see SWE-bench-style leaderboards adopt the same failure taxonomy instead of collapsing everything into one pass rate. A model that fails by missing a requirement needs a different review process around it than one that fails by breaking behavior it already had.