🚀 New: chi (χ) — an open-source autoresearch harness for fleets of LLM coding agents. Read the announcement.
Kuldeep Pisda
A Targeted tofu plan Said '2 to add'. The Unscoped Apply Replaced Both Production Servers

A Targeted tofu plan Said '2 to add'. The Unscoped Apply Replaced Both Production Servers

A published postmortem from 28 October: a plan run with -target showed 2 to add, the apply ran without it and showed 5 to add, 3 to destroy. Why the two runs were different plans, what a saved plan file changes, and why delete-before-create turned a drift into a 40 minute outage.

5 min read

Contents

On 28 October a small project called event-junkie published a postmortem for a production outage of about 40 minutes. The change that caused it was meant to add two DNS records. The operator ran tofu plan -target=hcloud_zone_rrset.redirect and got 2 to add, 0 to change, 0 to destroy. Then they ran tofu apply without -target. The apply planned the whole stack, reported 5 to add, 3 to destroy, and replaced both production servers.

The postmortem’s own root cause is one sentence: the checked plan and the applied plan were two different runs. That is the part worth generalizing, because the same split exists in any workflow that reviews with one command and applies with another.

Why the two runs disagreed

-target narrows what OpenTofu plans. The plan docs describe it as focusing planning “only on resource instances which match the given address”, and say it is provided for exceptional circumstances and not recommended for routine operations. A targeted plan is a statement about a subset of the graph. It says nothing about the rest of the configuration.

In this case the rest of the configuration had drifted. According to the postmortem, user_data on the servers forced replacement, and the cloud-init/ directory had changed since the last apply. None of that appeared in the targeted plan, because nothing outside the DNS record was in scope. The unscoped apply found the drift and acted on it.

The postmortem notes that the same trap had hit staging before, and that it was described in prose in an infra/AGENTS.md file. Nothing in code prevented it. It also lists the plan summary as the only guard against a destructive apply, and the summary did show 3 to destroy before the operator confirmed. The postmortem does not say whether the operator read beyond that line, so I will not guess at that. The structural point stands either way: a confirmation prompt that shows a different plan from the one you reviewed is a weak control.

What a saved plan file removes

OpenTofu has a mechanism for exactly this gap. tofu plan -out=<file> writes the plan to a file, and tofu apply <file> executes it. The apply docs say two things that matter here:

  • OpenTofu takes the actions in the saved plan without prompting for confirmation.
  • When using a saved plan, you cannot specify any additional planning modes or options.

So with a plan file there is no second planning pass in which a flag can be forgotten, and the thing you reviewed is the thing that runs. The second bullet also closes a quieter failure: you cannot pass -target to the apply and have it mean something different from what the plan did.

That matches the postmortem’s action items: an infra/apply.sh that applies the checked plan file and refuses unasked replacements, and a runbook that says the same. The postmortem does not state whether those are done.

One caution from the plan docs: the file contains your full configuration and planned values, and sensitive data may be stored in cleartext unless plan encryption is enabled. Treat it as a secret, not a build artifact to attach to a pull request.

Why replacement became an outage

Drift forcing a replacement is a bad day. Drift forcing a replacement that fails halfway is the outage. The postmortem gives two contributing facts.

First, the replacement destroys the old server before creating the new one. That is the default behavior for an argument that cannot be updated in place: OpenTofu destroys the existing object and then creates the replacement. create_before_destroy flips the order. The postmortem says the servers hold fixed private addresses and Primary IPs, and that is a plausible reason flipping the order is not free here, since two servers cannot hold the same fixed address at once. I do not know how their configuration is written, so treat this as a constraint to check in your own setup rather than a recommendation.

Second, the replacement server types, cx33 and cx23, could not be placed. The postmortem says Hetzner showed both as unavailable in every eu-central location. With the old servers already destroyed, there was nothing to fall back to. Production was rebuilt on cpx32 and cpx22 instead, with the database volume reattached.

The data survived because it lived on a separate volume, and the Primary IPs survived, so DNS did not need to change. Several other things did not survive. Every Web Push subscription on production ended, because the key pair existed only in a Secret on the old cluster and the rebuild generated a new one. The log and metrics store started empty, and Hetzner snapshot backups of both servers were deleted along with them. The monthly cloud bill rises from €33.21 to €91.76 until production returns to the original types.

The guard that fixes the cause, not the mistake

The first fix is process: apply the plan file. The second is in code. The postmortem proposes ignore_changes = [user_data] so that user_data changes do not force replacement unless someone asks for a rebuild by name.

The lifecycle docs are precise about what that does. Ignored arguments are still considered when planning a create, and ignored when planning an update. So a brand new server still gets its user_data, and an existing one stops being replaced because the file changed. The cost is the one you would expect: a cloud-init change no longer reaches existing servers on its own, which is why the postmortem frames it as rebuilding only when asked.

The two controls fail differently. The plan file protects against a wrong command. ignore_changes protects against a correct command that touches something nobody was thinking about. Only the second still helps if someone later bypasses the wrapper script.

Smaller findings after recovery

Both rebuilt nodes booted without patch monitoring: the hardening script stopped before installing node_exporter, but cloud-init reported success. About 30 minutes of the recovery went to manual steps (firewall, tunnel, kubeconfig, 14 Secrets, a backup credential), several of which were missing from the runbook. If you operate a cluster you rebuild rarely, those two items are the checklist: a script that exits early but reports success, and a recovery path that lives in someone’s memory.

Sources