Contents
Anthropic’s postmortem on three overlapping infrastructure bugs that degraded Claude’s output quality between early August and mid-September contains a detail worth sitting with: the worst of the three bugs wasn’t introduced by a mistake. It was introduced by a fix. Specifically, by deleting a workaround that had been quietly doing its job for eight months, because the engineers who wrote the replacement believed they had solved the problem the workaround existed for. They had solved a problem. Not the one they thought.
The workaround, and what it was actually hiding
Claude’s TPU serving stack computes next-token probabilities in bf16, a 16-bit float format, but the TPU’s vector processor is fp32-native. To save time, the XLA compiler can silently promote some of those bf16 operations to fp32 during optimization, a behavior gated by a flag called xla_allow_excess_precision, which defaults to true. When some operations in the sampling pipeline run at bf16 and others get auto-promoted to fp32, two code paths that are supposed to agree on which token has the highest probability can disagree, because they’re not doing the same arithmetic. In December 2024, this mismatch was observed as a specific, narrow symptom: at temperature zero, the model would occasionally drop the single most probable token from consideration entirely. Anthropic shipped a workaround and moved on.
The workaround held for close to two years. In August, engineers rewrote the sampling code to address precision issues more broadly and to clean up how probabilities near the top-p cutoff were handled. As part of that rewrite, they removed the December 2024 workaround, on the belief that the new code fixed its root cause properly instead of papering over it.
It didn’t. The workaround wasn’t just fixing the temperature-zero symptom. It was also masking a separate, deeper bug in approx_max_k, the approximate top-k operation Claude’s TPU stack uses to cheaply find the highest-probability tokens during sampling instead of running an exact sort across every chip holding a shard of the vocabulary. That approximation is supposed to trade a little accuracy in the low-probability tail for speed, a trade that shouldn’t touch output quality. Under certain batch sizes and model configurations, it was returning outright wrong results, sometimes dropping the highest-probability token instead. The December workaround happened to route around this too, which is why nobody had seen it: the symptom it was built for and the symptom it was accidentally suppressing looked identical from the outside, a token that should have been chosen wasn’t.
Once removed, the real bug surfaced, and it was worse to pin down than the one it replaced. Anthropic’s engineers described its behavior as inconsistent in a way that broke normal debugging assumptions: it depended on what operations ran immediately before or after it and on whether debugging instrumentation was even attached. The same prompt could succeed on one request and fail on the next, on the same infrastructure. A reproducer that failed reliably on TPU returned correct results when run on CPU, which is usually a sign that a bug lives in how an operation gets compiled for a specific accelerator rather than in the algorithm itself, and that’s where it turned out to be: a latent bug in the XLA:TPU compiler’s handling of approx_max_k, triggered by the August rewrite but not caused by it.
The fix Anthropic shipped wasn’t a patch to the approximation. It was dropping the approximation. They switched to exact top-k, which no longer carries the performance penalty it once did, and standardized more of the pipeline on fp32. Model correctness won the tradeoff outright: they call it non-negotiable and accepted the efficiency cost.
A second bug from the same class: sticky routing turns rare into personal
A different bug in the same window shows the same shape from another angle. On August 5, a routing error started sending a small fraction of Sonnet 4 requests, about 0.8%, to servers provisioned for the upcoming 1M-token context window instead of standard short-context servers. Wrong server pool, degraded response, low blast radius. Then on August 29, a routine load-balancing change unrelated to that bug increased how much short-context traffic got misdirected the same way. At the worst hour, on August 31, 16% of Sonnet 4 requests were affected, and roughly 30% of Claude Code users who made requests in that window had at least one message hit the wrong server type.
The mechanism that turned a percentage into a user experience was routing stickiness: once a session got pinned to the wrong server pool, its follow-up requests tended to stay pinned there too. A low base rate doesn’t protect you if the assignment is sticky, because the bug doesn’t sample a fresh coin flip per request, it samples once per session and then repeats. If you operate anything with session affinity and any per-request failure mode, however rare, assume affected sessions stay affected until something explicitly re-routes them. Aggregate rates undersell what the unlucky users actually experience.
Why the evals didn’t catch either one
Both bugs share a detection story worth taking seriously if you run any kind of automated quality gate. Anthropic’s benchmarks and canary rollouts didn’t catch the degradation because, in their words, Claude “often recovers well from isolated mistakes.” A model that occasionally drops the best token or gets routed to the wrong context window doesn’t necessarily fail a benchmark question outright, it just answers slightly worse, in a way that a scored eval suite built around correctness on fixed tasks isn’t shaped to detect. Layered on top of that, Anthropic’s own privacy controls limit engineer access to user conversations that weren’t explicitly reported as feedback, which is the right default and also meant the team couldn’t freely go pull the exact failing transcripts to reproduce the bug once reports started coming in.
The actionable version of this, for anyone shipping model-quality or correctness-sensitive infrastructure: a benchmark that scores end-to-end task success is a lagging, noisy signal for this class of bug. What worked here was building detection for the specific symptom, unexpected-character output tests for the corruption bug, a reproducer isolated down to a minimal compiler-level test case for the top-k bug, not waiting for aggregate benchmark scores to move enough to notice.
The decision this actually changes
The specific lesson isn’t “don’t delete workarounds.” It’s that a workaround you don’t fully understand is evidence, not just an eyesore. If you inherit one, the obligation before removing it isn’t to convince yourself your replacement fixes the bug you know about, it’s to write a regression test that pins down the exact original symptom and keep that test in the suite through the change, so a hidden second failure mode gets caught by CI instead of by a month of user reports. The Anthropic team’s own fix for the deeper bug wasn’t cleverness, it was giving up the approximation and eating the performance cost once correctness was on the line. That trade is available to you too, more often than the original performance-motivated choice made it look.
Sources
[1] https://www.anthropic.com/engineering/a-postmortem-of-three-recent-issues: Anthropic’s engineering postmortem on the three infrastructure bugs, including the XLA:TPU compiler bug, the context-window routing error, and the TPU sampling misconfiguration