Contents
Anthropic published a post-mortem this week on a service almost nobody outside its CI team had heard of: the test-selection system that decides which tests run on which pull request. It broke three times in six months. Each fix bought less runway than the last, 70 days, then 29, then under a day, before the team gave up patching and rebuilt the thing from a different set of assumptions. The rebuild took three weeks. The interesting part isn’t that Claude wrote a lot of the code involved, it’s that the failure mode wasn’t an outage. The service kept responding the whole time. It just started answering with stale, silently wrong data, which is a worse class of bug than a crash because nothing pages you for it.
What test impact analysis actually does
Running the full test suite on every pull request is the naive default, and it works until it doesn’t: gates get slower, more expensive, and eventually so noisy that engineers stop trusting a red build. Test impact analysis (also called test selection) is the standard fix, used widely enough that there’s a vendor category built around it: instead of running everything, a service tracks which tests actually exercise which parts of the codebase and picks a relevant subset per change.
Anthropic’s version split into two deterministic pieces that have to stay in sync. A listener records results from every CI run as they complete. A selector reads that history and decides, for a given PR, which tests are worth running. The two only work together if the listener’s view of “what happened recently” doesn’t lag behind what the selector is querying.
The v0 implementation kept a running history per test and required a single writer to apply results in order, so that the history stayed correctly sequenced. That single-writer constraint is the detail that matters: it’s what made the whole service a stateful singleton, and a stateful singleton is exactly the thing that can’t be horizontally scaled without a redesign.
Three patches, three shrinking windows
According to Anthropic’s account, engineers ship on average 8x as much code per quarter now as they did from 2021-2025, Claude authors roughly 80% of it, and the number of tests across the codebase grew 10x with only a modest increase in headcount. Those are Anthropic’s own reported figures, not independently audited, but the resulting number that matters mechanically is the one downstream of all three: CI job volume rose 25x over six months. The service was not built for that curve.
October: buy a bigger machine. Doubling the cores running the listener was the obvious move, and everyone involved knew it would be temporary. It held for 70 days.
February: shard by package. The single-writer constraint didn’t actually require one writer for the whole service, it required one writer per package, since that’s the unit results need to stay ordered within. Splitting each package’s state into its own shard with a dedicated worker let the system parallelize without breaking the correctness guarantee. This bought 29 days.
March: daily restarts. The process was hitting its memory limit by mid-afternoon on most weekdays. The team found four bugs, tried swapping the memory allocator (no effect, because allocator tuning wasn’t the actual bottleneck), and ruled out live memory profiling because the service was already a struggling singleton under load. Restarting it daily was the fallback. It bought less than a day before the next failure.
The restart patch also surfaced the real cost of the design. When the listener fell more than an hour behind, which happened repeatedly, a large number of job results simply weren’t recorded. Anthropic is explicit that this didn’t mean untested code reached production; CI still ran on those PRs. What it meant was that the selector was making its “which tests matter” decision from stale history, and the failure direction happened to be conservative: it mostly caused already-flaky or broadly-failing tests to get selected again, over-inclusion rather than a dangerous under-inclusion that could let a real regression slip through unselected. That’s a fortunate default, not a designed-in guarantee, and it’s worth noticing that a test-selection system built the other way, one where staleness trends toward running fewer tests, would have shipped the exact same six months of load growth as a silent correctness regression instead of a wasteful one.
The redesign that actually held
The fix was not a bigger or cleverer patch to the singleton. It was removing the constraint that created the singleton in the first place. Anthropic gave the listener an in-memory data store and turned it into an append-only journal: any worker can process any incoming result, append it, and move on without holding state in process memory. A small separate consumer rolls that journal up into per-test history every few seconds, and the selector reads from the rolled-up view rather than the live stream.
This is a recognizable pattern, log-structured ingestion with an async materialized view downstream, closer to how event-sourced systems or Kafka-plus-consumer architectures decouple write throughput from read state than anything CI-specific. What’s notable is the cost curve. The redesign runs more expensively than the singleton did. It is also, per Anthropic, meaningfully easier to scale and to memory-profile, because there’s no longer a single stateful process to load-test in production. And it took three weeks for one engineer, which the post claims would have been closer to a quarter a year earlier.
The decision this actually changes
If your CI pipeline has anything resembling a gatekeeper service, a flaky-test tracker, a coverage-based selector, a merge-queue prioritizer, that keeps a single in-process source of truth for ordering or correctness, the lesson here isn’t “adopt AI coding tools.” It’s that the retrofit cost for moving that state out of the process and into an append-only store keeps dropping, while the cost of leaving it in place doesn’t, because PR volume from any team running agentic coding at scale climbs faster than a singleton’s headroom regardless of who or what wrote the code. Anthropic’s own advice, budget for 10-20x your current perceived load in a v0 design if you can afford it, and assume 25x within two quarters if you’re already growing this way, is aggressive, but it’s aggressive in a specific direction: assume the process boundary between “record what happened” and “decide what matters” is the one that breaks, and put a durable log there before the paging starts, not after the third patch stops holding.
The other detail worth carrying over even if you’re nowhere near this scale: audit what your test-selection or caching layer does when it falls behind, not just whether it can fall behind. A staleness bug that trends toward running extra tests is an inconvenience. One that trends toward skipping tests is a regression waiting for the one week nobody notices the lag.
Sources
[1] https://claude.com/blog/agentic-coding-is-straining-ci-heres-how-we-scaled-test-impact-analysis-at-anthropic: Anthropic’s account of the test-selection service’s failures and redesign [2] https://claude.com/blog/ai-ci-cd-on-call: Related Anthropic post on Claude Tag as CI/CD on-call first responder