Railway's Breach Shows Per-Tenant Keys Don't Limit Who Can Decrypt

Railway’s incident report for the October 21 attack, published November 10, contains a detail that should bother anyone who stores customer secrets with envelope encryption. Railway encrypted each environment’s variables with its own data encryption key. Those keys were wrapped by a root key that lives in an upstream cloud provider’s KMS and “cannot leave the key management service.” That is the textbook design. The attacker still decrypted variables for up to 1.85% of workspaces, about 100,000, because a single credential on one machine was allowed to ask the KMS to unwrap any of those per-environment keys.

Per-tenant keys limit how much a stolen data key exposes. They do nothing about a principal that is authorized to unwrap every tenant’s key. In this design the blast radius is set by decrypt authority, not by key count.

The report also shows a second, quieter secrets store that most teams don’t think of as one: the workflow engine’s history.

The two secret paths#

Everything below comes from Railway’s report. Timestamps are UTC.

Path 1: build history in Temporal. At 07:28 on October 20 the attacker partially escaped filesystem isolation on Railway’s multi-tenant build VMs and read a credential the build servers used to push images to the registry. Railway’s build system ran on a self-hosted Temporal cluster, and its tasks carried a repo-scoped GitHub token, “the environment variables and secrets passed to the build,” and build metadata. The stolen credential “was intended to authorize only work needed to complete builds, such as registry pushes,” but the Temporal instance “accepted it as a trusted client.” Using client APIs, the attacker read the stored histories of 353 builds, sealed variables included, between 13:50 and 15:30 on October 20. The same credential could fetch repository snapshots used for build caching, which gave the attacker a copy of Railway’s internal monorepo.

Path 2: the variable store. With the monorepo in hand, the attacker found “an erroneously defined OIDC binding configuration” in one cloud environment and, at 03:26 on October 21, obtained a short-lived CI/CD cloud credential through it. That credential could retrieve two internal secrets: a longer-lived credential that, combined with internal VPC access, could read encrypted variables from a production database, and an SSH key the CD system used to push binaries. The attacker used the SSH key to reach a fleet orchestration machine. That machine had on disk a longer-lived credential “that allowed decryption of encrypted variables, which this machine needed to provision customer workloads onto other nodes in the fleet,” and it sat on the VPC with the variables database.

From 04:07 to 05:21 the attacker pulled encrypted envelopes, “selecting rows where the variable name matched attacker-provided regular expressions,” prioritizing AI provider keys and environments with many variables. From 04:42 to 07:14 they used the orchestration credential against the KMS to decrypt what they had pulled. Railway disabled all cloud credentials and the decryption service at 07:14.

Why the per-environment keys didn’t contain it#

Envelope encryption splits the problem in two. Data keys encrypt data; the root key encrypts data keys. Stealing a database dump gets you ciphertext and wrapped keys, both useless without the root key. Stealing one unwrapped data key gets you one environment.

What it does not do is decide who may call Decrypt on the root key. KMS treats every caller with that permission the same. If the orchestrator legitimately needs to start any customer’s workload, the simplest implementation gives it permission to unwrap any environment’s key, and anyone holding the orchestrator’s credential inherits that. The KMS root key never left the KMS. It didn’t need to.

Railway’s report says its logs “identify a subset of environments, not individual variables, so that figure is an upper bound.” That follows from the same design. Once the attacker unwrapped an environment’s data key, every variable in that environment was readable, so the only reliable accounting unit is the environment.

What partitioning decrypt authority looks like#

KMS products have a primitive for this, and it’s commonly misunderstood. In AWS KMS, an encryption context is a set of non-secret key-value pairs used as additional authenticated data: “the same encryption context is required to decrypt the data.” You can then condition kms:Decrypt on it. AWS’s documented example:

{
  "Effect": "Allow",
  "Principal": { "AWS": "arn:aws:iam::111122223333:role/RoleForExampleApp" },
  "Action": "kms:Decrypt",
  "Resource": "*",
  "Condition": {
    "StringEquals": { "kms:EncryptionContext:AppName": "ExampleApp" }
  }
}

Wrapping each environment’s data key with environment_id in the context binds the wrapped key to that environment, so a row copied into another environment’s slot fails to decrypt. It also puts the environment ID in plaintext in every KMS audit log entry, which turns a regex-driven bulk decryption into something you can alert on: one principal decrypting for hundreds of distinct environments in an hour does not look like scheduling.

It does not, by itself, stop the Railway attack. The caller supplies the context. A policy that lets the orchestrator decrypt “for any environment” still lets a thief decrypt for any environment. The condition only shrinks blast radius when the allowed values are narrow, and a fleet scheduler’s legitimate scope is the whole fleet.

That leaves an architectural fix rather than a policy one, which is roughly what Railway describes doing: “no service other than the secrets service can get access to keys that allow decryption of data encryption keys. Each calling service is restricted to decrypting only the set of credentials it needs to operate,” with every encrypt and decrypt “centrally audit logged.” The useful property is that the secrets service can check a request against independent state, for example “is there a pending placement of environment X on node Y right now,” instead of trusting the caller’s word about which environment it wants. A stolen orchestrator credential then gets secrets for workloads actually being scheduled, at the rate they are scheduled, rather than a decryption oracle.

The workflow engine is a secrets store#

Path 1 needed no KMS at all. Temporal’s data encryption docs are explicit that the service “persists data from your Workflow Executions, including inputs, outputs, and results,” and encryption is opt-in: you “use a Payload Codec to encrypt payloads before they reach the Temporal Service.” With it, “data exists unencrypted only on the Client and the Worker process.” Without it, every secret you pass as a workflow or activity argument is retained in history for as long as your namespace retention keeps it, and anything the cluster accepts as a client can read it.

Two things went wrong together. The build secrets sat in history in a readable form, and the cluster’s authorization treated a registry-push credential as a full client. Either one alone would have made the 353-build read much harder. The general lesson for any durable workflow engine, job queue or task table: arguments are stored, so pass a reference to a secret and resolve it inside the worker, or encrypt payloads with keys the server never holds. Also check what your cluster’s authorizer actually does with a credential that is valid but was minted for another purpose. “Authenticated” and “allowed to read other builds’ history” should not be the same check.

Notes on the isolation layer#

Railway had built a new builder that puts each build in its own ephemeral VM on top of the existing isolation, with a credential that can only push one specific OCI image. It was scheduled to start rolling out the week of October 26. After the first escape was patched, an internal hardening team “derived an additional exploit by modifying the original vector” within an hour, and Railway moved every customer to the new builder at once. The report says that cut-over drove build volume to “nearly two orders of magnitude above baseline” and storage load at least 10x, which is why builds stayed slow or failing for days after the attacker was out.

If your plan for a known-weak isolation boundary is a gradual migration, size the emergency version of that migration too. You may be forced to do it in a day.

What to check in a multi-tenant system#

• For each credential that can call KMS Decrypt, write down how many tenants it can decrypt for. If the answer is “all,” that credential is your real blast radius, whatever the key hierarchy looks like.
• Search workflow and job payloads for secret values. Temporal histories, Celery results, Sidekiq retries and outbox tables all persist arguments.
• Alert on the number of distinct tenants per decrypting principal per hour, not just on decrypt volume.

Railway’s report is unusually specific about mechanism, and the corrections it makes to its own preliminary timeline (decryption started at 04:42, not after 05:27; signups began October 18) are worth reading in full.

Sources#

• Railway, Incident Report: October 21st Security Attack (November 10, 2026)
• Railway, Preliminary Report: Oct 21st Security Incident (October 22, 2026)
• AWS KMS Developer Guide, Encryption context
• Temporal docs, Data encryption