Contents
The last chapter ended with a decision, not an action. We had outgrown the Wasabi-plus-Cloudflare stack that had saved us a small fortune, and I had picked Bunny.net — Bunny Storage for the files, Bunny CDN in front — as the place our 3.5 TB of media was going to live. Deciding is the easy part. The hard part is the sentence I had to say next: we are going to move roughly 3.5 terabytes of video from one object store to another, and at no point during that move can a single one of our 30,000 users notice.
This chapter is about how you move a mountain out from under people who are standing on it.
What “live migration” actually means
Let me define the thing precisely, because the phrase gets thrown around loosely. A live data migration is moving data from an old store to a new store while the system is still serving real traffic against that data the entire time — reads and writes never stop. Nobody gets a maintenance page. Nobody’s video stalls. The old store and the new store are both real, both correct, and both in play until the moment you decide otherwise.
Contrast that with the migration everyone reaches for first: take the app down, copy everything, point the app at the new place, bring it back up. That is a big-bang migration, and for a lot of systems it is completely fine. For us it was a non-starter, and it is worth being honest about why, because “we can’t have downtime” is a lazy reason on its own.
An object store is just a service that holds files — “objects” — and hands them back by key. Think of it as a giant hard drive you talk to over HTTP. Copying 3.5 TB between two of them is not instant. Even at a healthy sustained 200 Mbps of copy throughput, 3.5 TB is on the order of a day and a half of continuous transfer — and object storage rarely gives you clean sustained throughput across millions of small HLS chunks. (Recall from earlier chapters that we don’t store one big file per lesson; transcoding chops each lesson into thousands of small few-second .ts chunks across several quality levels.) At cutover we were holding ~3.5 TB across ~4.2 million objects — the vast majority of them tiny .ts segments, plus the .m3u8 playlists that stitch them together. So a big-bang copy would mean either a day-plus of downtime, or — worse — a “quick” copy that silently misses every file uploaded during the copy. For a platform where creators upload 150+ videos a day, the data is a moving target. You cannot photograph a moving target.
There is a second reason the two stores could not simply be swapped, and it is the one that shaped every line of code below. Wasabi is S3-compatible; Bunny is not S3 at all. Wasabi speaks the S3 API — you talk to it with boto3 as long as you set endpoint_url and force path-style addressing. Bunny Edge Storage speaks its own flat HTTP API: PUT/GET/DELETE against https://storage.bunnycdn.com/<zone>/<path> with an AccessKey header. That mismatch is exactly why, in the prior chapter, MediaConvert refused to write to Wasabi — it wanted true S3 semantics Wasabi doesn’t fully honour. Here it means the copy can’t be a single aws s3 sync: the read side is S3, the write side isn’t, and nothing off-the-shelf bridges them for free.
So the question was never how do I copy 3.5 TB. It was how do I copy 3.5 TB while it keeps changing underneath me, and prove the copy is perfect before I trust it.
The four moving parts
Live migrations are built from four ideas. None of them is exotic; the discipline is in doing all four and not skipping the boring one.
Dual-write. From a chosen moment forward, every new write goes to both stores at once. Upload a file, it lands in Wasabi and Bunny. This freezes the problem: from that moment, the new store is never missing anything new. All that’s left to move is the past.
Backfill. A background job walks everything that existed before dual-write began and copies it across. While users stream, a worker is quietly shovelling the historical 3.5 TB from Wasabi into Bunny, object by object. Dual-write handles the future; backfill handles the past. Between them, they converge on “both stores hold everything.”
Checksum / parity verification. This is the boring one, and it is the whole game. A checksum is a short fingerprint computed from a file’s bytes — feed the same bytes in, get the same fingerprint out; change one byte, the fingerprint changes completely. Parity verification means: for every object, compute its fingerprint on both stores and confirm they match. Same key, same size, same checksum, both sides. Only when parity holds across the entire keyspace do you actually believe the new store is a faithful twin of the old one. Copying is optimistic. Verification is what lets you stop praying.
Atomic cutover. The cutover is the instant you switch reads from the old store to the new one. Atomic means it happens all at once, cleanly, with no in-between state where some users read old and some read new inconsistently — and, crucially, it can be reversed just as cleanly. Done right, cutover is not a migration event at all. It is flipping one config value.
Here is the whole shape in one picture.
sequenceDiagram
participant U as Uploader
participant App as Django
participant W as Wasabi
participant B as Bunny
participant J as Backfill job
participant P as Parity check
Note over App: Dual-write phase begins
U->>App: Upload new file
App->>W: Write object
App->>B: Write same object
App-->>U: Ack after both succeed
J->>W: List all historical objects
J->>B: Copy any that are missing
P->>W: Read size and checksum
P->>B: Read size and checksum
Note over P: Compare - every object must match
Note over App: Cutover - flip read source to Bunny
U->>App: Request a file
App->>B: Serve from BunnyThe fork in the road: two ways to move
I considered exactly two strategies, and the difference between them is entirely about when the new store becomes trustworthy.
Option A — big-bang copy-and-switch. Freeze writes (or take a maintenance window), run one large copy, verify loosely, repoint the app, unfreeze. Simple to reason about. Simple to explain. And wrong for us on two counts: the freeze window at our upload rate was unacceptable, and — the deeper problem — the switch is a cliff. You point at the new store and immediately depend on it being perfect. If the copy missed even a handful of chunks, the first person who discovers it is a user whose lesson buffers forever at the 12-minute mark. The failure mode is discovered in production, by a human, on the one path you can’t observe well.
Option B — keep both in sync until cutover. Dual-write, backfill, verify continuously, and only flip reads once parity is proven and has stayed proven. More moving parts. More code. But the new store earns trust gradually and measurably, and the flip happens after you already know both sides are identical. The switch stops being a leap of faith and becomes a formality.
I chose Option B, and the reason is a principle I keep coming back to: the goal of a risky migration is to make the risky moment boring. Big-bang concentrates all the risk into one irreversible instant. Dual-write-and-verify spreads that risk out over days of observable, correctable, reversible steps, so that by the time you flip, there is nothing left to be nervous about.
How we actually built it
The single thing that made this tractable was a decision made chapters earlier: all media access already went through one storage abstraction — a thin Django storage backend, not raw SDK calls scattered across the codebase. Because there was exactly one door to the object store, I could change what happened behind that door without touching a single view, serializer, or Celery task. If storage calls had been sprinkled everywhere, this migration would have been a month of find-and-replace and a prayer. It was a few hundred lines in one place instead.
The Wasabi side was a stock django-storages S3 backend, with the two settings that S3-compatible-but-not-S3 stores always need — an explicit endpoint_url and path-style addressing, because Wasabi doesn’t do virtual-host-style buckets the way AWS does:
# settings/storage.py
from botocore.config import Config
WASABI_CLIENT_CONFIG = Config(
signature_version="s3v4",
s3={"addressing_style": "path"}, # virtual-host style breaks on Wasabi
)
STORAGES = {
"default": {"BACKEND": "media.storage.DualWriteStorage"},
"legacy": {
"BACKEND": "storages.backends.s3.S3Storage",
"OPTIONS": {
"bucket_name": "happy-thoughts-media",
"region_name": "ap-southeast-1",
"endpoint_url": "https://s3.ap-southeast-1.wasabisys.com",
"access_key": os.environ["WASABI_ACCESS_KEY"],
"secret_key": os.environ["WASABI_SECRET_KEY"],
"client_config": WASABI_CLIENT_CONFIG,
},
},
# Bunny is not S3 — it's a flat HTTP PUT/GET API, so it gets a
# hand-rolled backend (BunnyStorage) rather than the s3 one.
"new": {
"BACKEND": "media.storage.BunnyStorage",
"OPTIONS": {
"zone": "happy-thoughts",
"host": "https://storage.bunnycdn.com",
"cdn_base": "https://cdn.example.org",
"password_env": "BUNNY_STORAGE_PASSWORD",
},
},
}
The dual-write itself was deliberately unclever. Write to the old store first — it’s the source of truth until cutover — then mirror to the new one:
class DualWriteStorage(Storage):
"""Writes to the legacy store and mirrors to the new one.
Reads follow a flag so cutover is a config change, not a deploy."""
def __init__(self):
self.legacy = storages["legacy"]
self.new = storages["new"]
def _save(self, name: str, content) -> str:
# Legacy store stays the source of truth during migration.
saved_name = self.legacy.save(name, content)
try:
content.seek(0)
self.new.save(saved_name, content)
except StorageError:
# Never fail a user's upload because the *mirror* hiccuped.
# The backfill job will heal this key later.
log.warning("mirror_write_failed", key=saved_name)
return saved_name
def url(self, name: str) -> str:
store = self.new if read_from_new_store() else self.legacy
# During dual-write, a miss on Bunny falls back to Wasabi so a
# not-yet-backfilled chunk never 404s a live stream.
if store is self.new and not self.new.exists(name):
return self.legacy.url(name)
return store.url(name)
Two design choices in there carry the whole safety story. First, a failed mirror-write never fails the user’s upload — the legacy store is still the truth, and the backfill sweep will catch any key the mirror dropped, so a Bunny hiccup is a footnote, not an incident. Second, reads are gated by a runtime flag — read_from_new_store() — not a code path. That flag is the cutover switch, and it lives in config, so flipping it needs no deploy and un-flipping it is instant.
The backfill: paginate, hash, push, verify
The backfill was a throttled Celery worker on its own named queue — the queue-per-workload setup from chapter 7 earning its keep, because a day-and-a-half backfill crawl must never sit in front of a payment callback. Same RabbitMQ, different queue, different worker pool:
# celeryconfig.py
task_routes = {
"media.tasks.backfill_key": {"queue": "backfill"},
"media.tasks.verify_key": {"queue": "backfill"},
"payments.tasks.*": {"queue": "payments"}, # never blocked by us
}
task_acks_late = True # re-deliver a job if a worker dies mid-copy
worker_prefetch_multiplier = 1 # don't let one worker hoard the queue
# The backfill fleet — 16 concurrent copiers, isolated from payments.
celery -A app worker -Q backfill -c 16 --prefetch-multiplier=1
celery -A app worker -Q payments -c 4 # untouched, undisturbed
The driver used a boto3 paginator to walk the Wasabi keyspace — you never list millions of objects in one call, you page through 1,000 at a time — and a thread pool to copy many objects at once. acks_late plus idempotency is what makes “just start it again” a safe sentence: a job that dies mid-copy gets redelivered, and a key that’s already correct is skipped, so re-running never duplicates or corrupts anything.
import hashlib
import httpx
from concurrent.futures import ThreadPoolExecutor, as_completed
BUNNY = "https://storage.bunnycdn.com/happy-thoughts"
def backfill_key(key: str, size: int) -> str:
if _already_present(key, size): # idempotent: HEAD Bunny, match size
return key
body = wasabi.get_object(Bucket="happy-thoughts-media", Key=key)["Body"].read()
digest = hashlib.sha256(body).hexdigest().upper()
# Bunny verifies the upload against this header and rejects a corrupt
# body with 400 — integrity is enforced at the moment of write.
resp = httpx.put(
f"{BUNNY}/{key}",
content=body,
headers={
"AccessKey": os.environ["BUNNY_STORAGE_PASSWORD"],
"Checksum": digest,
"Content-Type": "application/octet-stream",
},
timeout=60.0,
)
resp.raise_for_status()
return key
def run_backfill(prefix: str = "", workers: int = 16) -> None:
paginator = wasabi.get_paginator("list_objects_v2")
with ThreadPoolExecutor(max_workers=workers) as pool:
futures = {}
for page in paginator.paginate(Bucket="happy-thoughts-media", Prefix=prefix):
for obj in page.get("Contents", []):
fut = pool.submit(backfill_key, obj["Key"], obj["Size"])
futures[fut] = obj["Key"]
for fut in as_completed(futures):
key = futures[fut]
try:
fut.result()
except Exception as exc: # one bad key must not kill the sweep
log.warning("backfill_failed", key=key, error=str(exc))
verify_key.delay(key) # requeue for re-copy + re-check
The multipart-ETag gotcha
Then parity verification, the part I refused to shortcut. The obvious cheap check is: compare the ETag both stores report and call it a day. That works right up until it silently doesn’t, and the reason is a trap that has burned everyone who’s done this once:
An S3/Wasabi ETag is only the object’s MD5 when the object was uploaded in a single part. Anything uploaded via multipart — which is everything our transcoder wrote for larger renditions — gets an ETag that looks like this:
"9b2cf535f27731c974343645a3985328-42"
That trailing -42 means “42 parts.” The hex in front is not the MD5 of the file — it’s the MD5 of the concatenated MD5s of each part, then hashed again. Its value depends on the part size the uploader happened to use, which Bunny neither knows nor reproduces. So comparing a Wasabi multipart ETag against anything Bunny computes is comparing two unrelated numbers and feeling reassured for no reason. For pure S3-to-S3 legs earlier in the saga I could lean on rclone check (which understands S3 ETags and part sizes), e.g.:
# Both remotes are real S3 endpoints, so rclone can compare ETags/part sizes.
rclone check wasabi:happy-thoughts-media s3backup:happy-thoughts-media \
--checksum --one-way --missing-on-dst /var/log/parity-missing.log
Bunny changes that. It isn’t an S3 remote — rclone check has no Bunny backend that reproduces S3 part-size math, and Bunny exposes no comparable checksum on read anyway: not an MD5 ETag, and not even the SHA256 I sent in the upload’s Checksum header (Bunny validates that header at write time and then discards it). So there is no metadata shortcut in either direction, single-part or multipart. The only honest fingerprint is one I compute myself, from the bytes, on both sides. Parity verification therefore checked size first (cheap, catches truncated writes instantly), and for the content check it always re-hashed the actual bytes — recomputing the same SHA256 on Wasabi’s object and on Bunny’s, and comparing those:
def verify_key(key: str) -> bool:
head = wasabi.head_object(Bucket="happy-thoughts-media", Key=key)
src_size = head["ContentLength"]
bunny_head = httpx.head(f"{BUNNY}/{key}", headers=_bunny_auth())
if bunny_head.status_code != 200:
return False
if int(bunny_head.headers["Content-Length"]) != src_size:
return False # size mismatch: truncated copy
# No usable checksum on either side to compare cross-store: Wasabi's ETag
# may be a multipart hash (not an MD5), and Bunny surfaces nothing on read.
# So re-hash both bodies with the same SHA256 the backfill uploaded with.
src_hash = _sha256(wasabi.get_object(Bucket="happy-thoughts-media", Key=key)["Body"])
dst_hash = _sha256(httpx.get(f"{BUNNY}/{key}", headers=_bunny_auth()).iter_bytes())
return src_hash == dst_hash
Every mismatch was logged with its key, fed back to the backfill queue to re-copy, and re-verified. I let it run until mismatches hit zero and stayed at zero across full sweeps. That “and stayed” is the point — one clean sweep can be luck; several clean sweeps across a still-live, still-being-written system is evidence.
One more safety net covered the window before parity was perfect, and it’s already baked into DualWriteStorage.url() above: during dual-write, a read routed to the new store falls back to the old one on a miss. If Bunny didn’t yet have a chunk, we served it from Wasabi instead of throwing an error.
flowchart TD
R[Read request] --> Q{Cutover flag on}
Q -->|no| WR[Serve from Wasabi]
Q -->|yes| BR[Try Bunny]
BR --> H{Object present}
H -->|yes| S[Serve from Bunny]
H -->|no| F[Fallback to Wasabi]The cutover that wasn’t an event
When parity had held clean for days, cutover was exactly the anticlimax I had designed for. read_from_new_store() reads a single row — one boolean in a settings table, cached in Redis — so the flip is one write, no deploy, and it takes effect on the next request across every process at once. There is no per-user rollout, no gradual percentage, no window where some users read Bunny and some read Wasabi: it is atomic precisely because it’s one shared value that every worker reads.
# The entire cutover. One row. Global. Reversible.
def set_read_source(*, use_new_store: bool) -> None:
StorageFlag.objects.update_or_create(
key="read_from_new_store",
defaults={"enabled": use_new_store},
)
cache.delete("flag:read_from_new_store") # force re-read on next request
# Cutover:
set_read_source(use_new_store=True)
# Rollback, if anything looks wrong — same call, other direction:
set_read_source(use_new_store=False)
I flipped it to true, and reads began coming from Bunny CDN. Then the verification pass I’d staged for the moment: a sampled read-through that pulled a few thousand live keys via the public CDN path (not the storage API — the thing users actually hit) and confirmed 200s and correct sizes. Dual-write kept running the whole time, so Wasabi stayed a live, current, byte-for-byte hot standby — which is what made rollback a non-event. My rollback plan was set_read_source(use_new_store=False): one row, instant, zero data loss, because both stores were still identical and still both being written. I watched Bunny’s cache-hit ratio climb and error rates stay flat, left the flag on through a full peak-traffic evening, and only then let Wasabi’s writes wind down, keeping it a while longer as a cold backup before decommissioning.
~3.5 TB and ~4.2 million objects moved. Zero downtime. Zero streaming interruptions. Zero support tickets. The single most consequential infrastructure change of the whole project was, from a user’s seat, the least eventful. Nobody watching a lesson that evening had any idea the file had changed continents.
The lesson
Migrate live by making the switch boring. The instinct on a scary migration is to over-prepare the big moment — the war room, the runbook, the held breath at 2 a.m. The better engineering move is to design the big moment out of existence. Keep both sides correct and continuously proven identical, so the cutover carries no risk that hasn’t already been measured and cleared. Dual-write removes the fear of missing new data. Backfill removes the fear of missing old data. Parity verification removes the fear that “copied” doesn’t mean “correct.” And a read flag makes the switch — and the un-switch — a single reversible config change. When you’ve done all four, the cutover isn’t a leap. It’s a shrug.
The corollary is the cheaper lesson, paid for chapters ago: the reason any of this was a few hundred lines instead of a rewrite is that every storage call in the system already went through one door. You buy the ability to do a calm migration long before you need it, by not scattering your dependencies in the first place.
With storage, CDN, transcoding, and now migration all finally at peace — no fires, no fair-use cliffs, no held breath — I could lift my head from the infrastructure for the first time in a long while and look at the product itself. And the most-requested, least-solved thing our Hindi-speaking users kept asking for was simple to say and hard to build: let me actually search for what I want to learn.