Contents
Control D’s paid DNS stopped answering queries at every location for 55 minutes on October 11, 2026. The postmortem, published October 20, puts the cause in a change that had shipped 25 days earlier and had not done anything since.
The shape of the failure is worth separating into three independent defects, because each one has a different fix and a different place in your own systems where it could be hiding.
A type change that nothing exercised
On September 16, Control D updated its API to add validation for an optional per-domain override on “services”, which are prebuilt entries such as streaming platforms, each backed by a list of domains. According to the postmortem, the update made the API save these overrides as integers. The DNS software reads them as strings.
Nothing broke, because nothing wrote one. On October 11 at 16:40 UTC an engineer added four domains to one service with an “allow” override. By Control D’s account this was the first save of that kind since the September change. The API stored the overrides as numbers and told every paid-plan DNS server to reload. By 16:42 no paying customer’s queries were being answered anywhere.
The useful observation is that the defect had an activation condition that was a rare product action, not time or load. A canary on the September deploy would have shown nothing. Control D says its tests did not check that API-saved data stayed in a format the DNS software could read. That is a contract between two components that lived only in people’s heads and in two codebases that happened to agree until one of them changed.
A startup check that converted a bad value into a restart loop
The reload failing is the first defect. The second is what the DNS software did with the unreadable value. Per the postmortem, it stopped instead of logging the problem and skipping the entry. Its startup safeguard was designed to refuse to start if any configuration could not be read, so each automatic restart hit the same stored values and failed again.
This is a defensible rule in isolation. A resolver that starts with half its configuration might answer queries wrongly, for example by failing to apply a block rule. Control D states that no query was answered incorrectly during the outage, which is the property that rule protects. The cost is that the unit of failure became the whole process, not the one service entry with the wrong type.
The remediation shows where they landed on that tradeoff. The DNS software now logs and sets aside unreadable service entries during reloads and at startup. That is a deliberate move from fail-closed to degrade-and-report, scoped to service entries. Whether it is the right trade depends on what the skipped entry does. A bad override on a streaming bundle is a different risk from an unparsable block rule, and the postmortem says a broader audit of configuration types in the DNS software is still in progress. I would not read the fix as “never refuse to start”. I would read it as “decide per config class what an unreadable value costs”.
Shared configuration with no staging
The third defect is why one save reached every server. Service domain lists are shared configuration delivered to all paid-plan DNS servers at once. The postmortem says configuration changes were not rolled out in stages, unlike software updates. Failover moved traffic toward healthy locations, but every location had the same stored values, so there was nothing healthy to move to.
This is a common asymmetry. Software releases get canaries, health checks and rollback. Data pushed by the same control plane often does not, even though it can crash the same binaries. The free tier staying up is consistent with it running on separate configuration, but the postmortem does not spell out that mechanism, so I will not claim it.
Detection and recovery
Timing from the postmortem, in UTC:
| Time | Event |
|---|---|
| 16:40 | Four overrides saved, reload pushed to all paid servers |
| 16:42 | No paid queries answered anywhere |
| 16:46-16:47 | First automated alerts, reporting individual servers and locations, not a full outage |
| 16:51-16:55 | Internal reports, on-call paged |
| 17:16 | Cause traced |
| 17:30 | Service saved again with the four overrides removed |
| 17:31-17:35 | Service restored at every location |
Two things stand out. The alerts described symptoms per server, so no alert said “paid DNS is at zero”, and Control D lists new paging rules for queries dropping to zero as a partial fix. And the cause was found 36 minutes after the change, with the repair taking 14 more minutes: removing four bad values and saving again. Service edits were frozen until both fixes were in place.
The company also acknowledges that it had no incident status page, that its network status page showed most locations healthy, and that many customers learned of the outage from Reddit or Discord.
The two controls this implies
Control D’s list of planned work includes validating API-written configuration against the DNS software’s reader before publishing, and staged rollout of configuration changes. Those map to the two places a mismatch like this can be stopped.
The write-path check runs the real parser from the consumer over what the producer is about to publish. It is the one that would have rejected the integer on October 11 and, with a test that saved a service override, would have failed the September 16 change itself. It requires that the reader is importable or callable from the control plane, which is an architectural constraint, not a lint rule.
The read-path check is the degrade-and-report behavior they have already shipped, plus the staged rollout still in design. It limits what a bad value can take down even when the write path let it through.
If you operate a system where one service writes configuration that another parses, a quick audit is to find every field the consumer treats as a typed value and ask two questions: what runs the consumer’s parser before the value is distributed, and what does the consumer do on a parse failure at startup, not only at reload. The second matters because a reload failure can usually keep the old config, while a restart has no old config to keep.
Control D also says it will start a re-architecture of its core DNS system later this month and review how its control plane and resolvers share state. Those are announced plans, not results.
Sources
[1] https://controld.com/blog/incident-october-11-2026/: Control D, “Incident Report: Control D DNS Outage on October 11, 2026,” October 20, 2026. All timings, causes and remediation items above come from this post and are Control D’s own account; I did not independently verify them.