Zeroing the Egress Bill: Wasabi, Cloudflare, and the Transcoder It Cost Us

Putting a cost-to-serve number next to daily active users told me exactly what the Happy Thoughts platform was spending on bandwidth. It did not make the spending stop. Once the 1,500 dollar day had my full attention, I did the obvious thing — went hunting for a cheaper way to put video in front of people — and the answer that eventually won was not a cheaper CDN. It was to stop paying for egress at all.

The half of the bill I hadn’t touched yet#

An egress bill is a multiplication with exactly two terms: how many gigabytes you serve, and the price per gigabyte. That is the whole equation, and it is worth staring at, because you attack a media platform’s delivery cost by pushing down one term or the other, and they are completely different projects.

By this point I had already done real work on the first term. We transcoded — turned each fat 1.2 GB HD original into an adaptive HLS ladder, so a phone on a thin Indian mobile pack pulled a light 360p or 480p rendition instead of the desktop-grade file. (Without that laddering, a single 20-minute lesson could eat a viewer’s entire daily data allowance; the ladder is what makes the platform usable at all.) That cut gigabytes-per-view hard. But it left the other term completely untouched.

And the other term was ugly for us specifically. Egress is the charge a cloud levies for moving bytes out of its network — off the origin, across the CDN, to the viewer’s phone. CloudFront prices egress by destination region, and India is one of the expensive ones: roughly 0.17 USD per gigabyte at the time, close to double what the same bytes cost served into the US or Europe. So even a lean HLS ladder, multiplied across a growing audience, multiplied by a price-per-GB set by geography I couldn’t change, still added up to a bill that grew every time the mission succeeded.

Put real numbers on it. A busy day around then pushed on the order of 9 TB out through CloudFront to Indian eyeballs. Nine thousand gigabytes times 0.17 is the roughly 1,500 dollar day that started this whole hunt — and that is just the CDN-to-viewer hop. There was a second, quieter egress charge underneath it: every time CloudFront had a cache miss it pulled the chunk from the S3 origin, and S3-to-CloudFront origin egress is billed too (a few cents a gigabyte). Two metered hops on the same bytes, both denominated in geography I couldn’t renegotiate.

Transcoding had bent the first term as far as it reasonably could without wrecking video quality. The lever I hadn’t pulled was the price per gigabyte. So I went to pull it.

Crash course: object storage, and what “S3-compatible” really means#

To follow the fix, two concepts.

First, object storage. It is not a filesystem with folders. It is a flat store of objects — a blob of bytes plus a key (a string like courses/42/720p/seg_003.ts) — grouped into buckets, and you talk to it over HTTP: PUT to write, GET to read. AWS’s version is S3, and because S3 got there first and got huge, its HTTP API became the de-facto standard for object storage. So many tools speak “the S3 API” that other vendors built their own storage that answers to the exact same calls.

That is what S3-compatible means, and it is the load-bearing idea in this chapter. A vendor like Wasabi is not AWS S3 — different company, different data centers, different backend — but it exposes the same API surface. Your code keeps using the S3 client it already has; you just point it at a different endpoint URL and hand it different keys. In practice, migrating our application off S3 was almost a one-line change:

# S3-compatible means the SAME boto3 client, aimed somewhere else.
s3 = boto3.client(
    "s3",
    endpoint_url="https://s3.ap-southeast-1.wasabisys.com",  # <- the whole trick
    aws_access_key_id=WASABI_KEY,
    aws_secret_access_key=WASABI_SECRET,
)
s3.upload_file("out_720p.ts", "ht-media", "courses/42/720p/seg_003.ts")  # just works

Every place our Django app read or wrote media went through one storage wrapper. Change the endpoint and the credentials in one config block, and the app was talking to Wasabi instead of S3, none the wiser. Hold onto how easy that felt — it is exactly the assumption that bites later.

The move: buy free egress with the Bandwidth Alliance#

Two things made Wasabi the target. It charges a flat per-terabyte storage rate with no egress fee and no per-request fee — the pricing model is deliberately the opposite of AWS’s meter-everything approach. And it is a member of the Bandwidth Alliance: a set of storage and cloud providers that agreed with Cloudflare to waive the egress fees between their networks. Data leaving a partner’s storage, bound for Cloudflare’s CDN, crosses that boundary for free.

Line up what that does to both terms of the bill along the whole delivery path:

• Origin to CDN. When Cloudflare needs a video chunk it doesn’t have cached, it pulls it from the Wasabi origin. Normally that origin pull is billable egress. Under the Bandwidth Alliance, Wasabi waives it — zero.
• CDN to viewer. A CDN keeps copies of your chunks at edge locations near users, so most requests never touch the origin at all; that buffer bar creeping ahead of the playhead is just cached chunks. Cloudflare, unlike CloudFront, does not meter delivery bandwidth on its plans. So the hop from edge to phone is not a per-GB line either.

Both egress hops that used to scale with our success — origin-to-CDN and CDN-to-user — collapsed to effectively nothing. Here is the before/after per gigabyte, which is the whole argument on one card:

Storage became a flat, forecastable line I could put in a budget and defend to a non-profit’s board. Delivery, the thing that had produced a 1,500 dollar surprise, stopped being a variable I feared.

flowchart LR
    Up[Upload MP4] --> S3s[S3 source bucket]
    S3s --> MC[AWS MediaConvert]
    MC --> S3m[S3 media bucket HLS]
    S3m --> CF[CloudFront]
    CF -->|paid egress to India ~0.17 per GB| U[Viewer]

The storage swap, for real#

The crash-course one-liner above is the idea; the actual swap lived in Django’s storage backend. We use django-storages on top of boto3, and the entire cutover was one STORAGES block. The load-bearing options are endpoint_url (points boto3 at Wasabi instead of the implicit AWS host) and addressing_style — Wasabi wants path-style URLs (s3.<region>.wasabisys.com/<bucket>), not the virtual-hosted style AWS defaults to:

# settings.py — every read/write of Happy Thoughts media flows through this.
STORAGES = {
    "default": {
        "BACKEND": "storages.backends.s3.S3Storage",
        "OPTIONS": {
            "bucket_name": "ht-media",
            "region_name": "ap-southeast-1",
            "endpoint_url": os.environ["WASABI_ENDPOINT"],   # https://s3.ap-southeast-1.wasabisys.com
            "access_key": os.environ["WASABI_ACCESS_KEY_ID"],
            "secret_key": os.environ["WASABI_SECRET_ACCESS_KEY"],
            "addressing_style": "path",        # Wasabi is path-style, not virtual-hosted
            "custom_domain": "cdn.example.org",  # emit URLs on the CDN host, not the Wasabi host
            "querystring_auth": False,         # public HLS chunks — no signed query params
            "default_acl": None,
        },
    },
    "staticfiles": {"BACKEND": "django.contrib.staticfiles.storage.StaticFilesStorage"},
}

custom_domain is the quiet hero: it makes storage.url(key) return https://cdn.example.org/courses/42/... — the Cloudflare hostname — so players fetch chunks through the CDN, never directly from the Wasabi origin. Cloudflare is a CNAME to the Wasabi bucket endpoint; the origin pull behind it is the hop the Bandwidth Alliance zeroes.

Then Cloudflare has to be told to actually cache video, because by default it’s conservative about caching a response it can’t prove is static. A Cache Rule in front of cdn.example.org does it — cache everything on the media paths, hold segments long, hold playlists briefly (a VOD .m3u8 never changes, but keeping its TTL short is cheap insurance against a bad publish):

# Cloudflare Cache Rule — "cache-everything" for HLS
When incoming requests match:
    (http.host eq "cdn.example.org" and http.request.uri.path contains "/media/")
Then:
    Cache eligibility  = Eligible for cache      # cache-everything, even without upstream cache headers
    Edge TTL           = 1 year   for *.ts       # immutable segments — content-addressed keys
    Edge TTL           = 60 s     for *.m3u8     # playlist can be re-pulled cheaply
    Origin Cache-Control = On

Content-addressed segment keys (the key changes whenever the bytes change) are what make a one-year segment TTL safe: a re-transcode writes a new key, so there is never a stale-segment problem to invalidate.

Moving the objects that already existed#

The config swap points new writes at Wasabi. It does nothing for the terabyte of renditions already sitting in S3. Those had to be copied across, and because both ends speak the S3 API, rclone does it in one command with two configured remotes (aws: → real S3, wasabi: → the Wasabi endpoint). It walks both sides, transfers only what’s missing, and verifies by checksum:

# One-time backfill: S3 -> Wasabi, verified by checksum, before flipping the config.
rclone sync aws:ht-media wasabi:ht-media \
  --transfers 32 --checkers 64 \
  --s3-upload-concurrency 8 \
  --fast-list --checksum --progress

I ran it before flipping endpoint_url, re-ran it to catch the delta written in between, then cut over and re-ran once more as a reconciliation pass — copy, then flip, then reconcile. (This backfill got a lot more interesting at 3.5 TB later in the series; ch18 goes deep on doing it without a maintenance window. Here it was a boring afternoon.)

flowchart LR
    Up[Upload MP4] --> FF[ffmpeg on our box]
    FF --> W[Wasabi bucket HLS]
    S3[Old S3 media] -->|rclone sync one time| W
    W -->|free origin pull| CFL[Cloudflare cache]
    CFL -->|no bandwidth fee| U[Viewer]

Why not the conventional fixes#

The conventional move, the one every AWS account manager will steer you toward, is to negotiate the bill down — a CloudFront committed-use plan or a savings bundle in exchange for a spend commitment. I turned it down for two reasons. It shaves a percentage; it does not change the shape. You are still paying per gigabyte, that per-GB line still climbs with every happy viewer, and India’s premium is still baked in. Worse, a commitment asks a donation-funded org to pre-commit forecasted bandwidth spend — to promise AWS a number I had already proven, one dollar-1500-day at a time, that I could not forecast.

I also considered the two other honest options and rejected them:

• Squeeze gigabytes-per-view harder. Drop bitrates further, add a lower rung. But I was already on that term, and past a point you are trading against video quality on the exact devices where the experience is already marginal. Diminishing returns, real cost to the product.
• Self-host delivery. A few cheap VPS boxes running nginx as a poor-man’s CDN. This is reinventing a CDN — edge locations near Indian users, TLS, cache invalidation, origin failover — for a one-engineer backend, to save money on a thing Cloudflare gives away. The operational tax dwarfs the win.

The unconventional call was to stop optimizing AWS’s delivery path and leave it entirely for a pairing whose price per gigabyte was structurally zero. Not a discount on egress — the deletion of egress as a line item.

So we did it. We copied the existing renditions from S3 across to Wasabi, flipped that one storage-config block so the app read and wrote Wasabi, and put Cloudflare in front as the CDN. The application never noticed. That was the entire point of S3-compatibility, and it worked exactly as advertised.

Right up until it didn’t.

The catch: “compatible” is not “the same,” and one service knew it#

Here is the assumption that felt free and wasn’t. Our code treated Wasabi as a drop-in for S3 because our code talks to storage through a generic S3 client where the endpoint is just a URL you can change. But not everything in the pipeline was our code, and not everything treats S3 as a URL you can repoint.

AWS MediaConvert — the managed transcoder I had moved to precisely because it absorbed our 150-videos-a-day upload floods without me owning any encoding hardware — is an AWS-native service. It does not take an endpoint_url for your storage. Its inputs and outputs are S3 URIs resolved inside AWS and authorized by an IAM role, not an endpoint-plus-access-key you can aim anywhere. There is, quite literally, no field in a MediaConvert job where you could name a Wasabi bucket:

# MediaConvert resolves storage INSIDE AWS, via an IAM role.
# There is no endpoint_url. The URIs must be real S3 or the job cannot run.
client.create_job(
    Role=MEDIACONVERT_ROLE_ARN,
    Settings={
        "Inputs": [{"FileInput": "s3://ht-source/raw/talk.mp4"}],
        "OutputGroups": [{
            "OutputGroupSettings": {
                "Type": "HLS_GROUP_SETTINGS",
                "HlsGroupSettings": {
                    # The one field that decides where the ladder lands.
                    "Destination": "s3://ht-media/courses/42/",   # must be s3://
                },
            },
            "Outputs": [ ... ],
        }],
    },
)

Put a Wasabi bucket in that Destination and the API rejects the job at validation time — it doesn’t even reach the encoder. The write path is the wall (MediaConvert can read an HTTPS input, but every output group’s destination is S3-only):

botocore.errorfactory.BadRequestException: An error occurred (BadRequestException)
when calling the CreateJob operation:
/outputGroups/0/outputGroupSettings/hlsGroupSettings/destination:
  Should match the required pattern ^s3:\/\/

There is no s3:// you can write that resolves to Wasabi. The scheme is s3://, but the resolver is AWS’s, keyed to your account and IAM role — a Wasabi bucket simply is not in the namespace it can see. The instant our media stopped living in real S3, MediaConvert had nothing it could legally write to. The managed transcoder wasn’t reconfigured or degraded. It was severed — cut off entirely, because the one storage it can speak to was the storage I had just walked away from to save on egress. Worth naming the second casualty too: leaving S3 also killed the native event integration — the s3:ObjectCreated notification that used to kick off a job the moment a raw upload landed. Wasabi’s bucket events don’t wire into EventBridge; that trigger had to be rebuilt in our own code.

flowchart LR
    Up[Upload MP4] --> FF[ffmpeg on our own box]
    FF --> W[Wasabi storage HLS]
    W -->|free egress Bandwidth Alliance| CFL[Cloudflare CDN]
    CFL -->|no bandwidth fee| U[Viewer]
    MC[AWS MediaConvert] -.->|blocked cannot write Wasabi| W

And I had been warned — by myself. When I first adopted MediaConvert, I’d noted in passing that it only reads and writes true S3, and filed it as harmless. It sat quietly as a load-bearing assumption for phases, until the day I changed something two hops away and it came due. The transcoding pipeline fell back to what it was before the managed service ever existed: plain ffmpeg, on our own machines, writing straight to Wasabi with the boto3 client from earlier.

The fallback wasn’t conceptually hard — it’s the same HLS ladder MediaConvert had been producing, just expressed as an ffmpeg invocation. The detail that matters, and the one people get wrong, is keyframe alignment across renditions. For a player to switch from 360p to 720p mid-stream, the segment boundaries have to line up on every rung — which means every rendition must cut a keyframe at the same wall-clock instant. You force that with a fixed GOP and no scene-cut keyframes: -g 48 -keyint_min 48 -sc_threshold 0 (a keyframe every 48 frames = 2s at 24fps), so 6-second segments are always switchable:

# CPU fallback: one raw MP4 -> a 3-rung switchable HLS ladder on Wasabi.
ffmpeg -i talk.mp4 \
  -filter_complex "[0:v]split=3[v1][v2][v3]; \
    [v1]scale=w=640:h=360[v360]; \
    [v2]scale=w=842:h=480[v480]; \
    [v3]scale=w=1280:h=720[v720]" \
  -map "[v360]" -c:v:0 libx264 -b:v:0 800k  -maxrate:v:0 856k  -bufsize:v:0 1200k \
  -map "[v480]" -c:v:1 libx264 -b:v:1 1400k -maxrate:v:1 1498k -bufsize:v:1 2100k \
  -map "[v720]" -c:v:2 libx264 -b:v:2 2800k -maxrate:v:2 2996k -bufsize:v:2 4200k \
  -map 0:a:0 -map 0:a:0 -map 0:a:0 -c:a aac -b:a 96k -ac 2 \
  -preset veryfast -g 48 -keyint_min 48 -sc_threshold 0 \
  -hls_time 6 -hls_playlist_type vod -hls_segment_type mpegts \
  -master_pl_name master.m3u8 \
  -var_stream_map "v:0,a:0 v:1,a:1 v:2,a:2" \
  -hls_segment_filename "out/%v/seg_%03d.ts" out/%v/index.m3u8

Correct, and completely beside the point at 150 uploads a day. libx264 on a -preset veryfast CPU core encodes a 20-minute talk at roughly real-time — call it 20 minutes of wall clock per video, three renditions deep. MediaConvert had been chewing through the daily flood in parallel across AWS’s fleet; a CPU box does them one core at a time and falls hours behind by mid-morning. The ladder was right. The throughput was a country mile short — which is the problem the next chapter has to solve.

The lesson: every infra swap has a blast radius#

The transferable idea is a discipline I now apply before any infrastructure swap, and it is not “don’t leave AWS.” It is: before you cut a dependency, map its blast radius — including the parts two hops away that you weren’t thinking about.

The change I was reasoning about was storage and CDN. The blast radius of that change reached a transcoding service I had mentally filed under a different problem entirely, because that service had a hidden, single-vendor tie to the exact thing I was replacing. The failure wasn’t leaving S3; leaving S3 was correct and it made delivery affordable for a non-profit that genuinely needed it. The failure was not asking one question upfront: who else reads or writes this bucket, and are any of them locked to this specific vendor rather than to the interface? Anything answering “locked to the vendor” is inside the blast radius, whether it feels related or not.

S3-compatibility is a real gift — it made our application migration a one-line change. But “compatible” is a statement about an interface, and the moment a component depends on the provider instead of the interface, compatibility stops protecting you. MediaConvert didn’t want the S3 API. It wanted AWS.

I’ll be honest about one more string attached, because it matters later: “free” egress under the Bandwidth Alliance came with a fair-use expectation — egress roughly in proportion to what you store. At our size then it was a non-issue, pure upside. Much further down this road, at a scale I couldn’t yet see, that expectation would become its own reckoning. But on the day I made this move, it was exactly the right trade: delivery went from a bill that punished growth to a flat storage line, and the platform could afford its own audience again.

Which left me with the bill mostly solved and the pipeline half-broken. I had cheap, effectively-free delivery — and, once again, no fast way to turn 150 uploaded originals a day into HLS ladders. MediaConvert was gone, ffmpeg on our CPU workers couldn’t come within a country mile of that throughput, and I owned exactly zero GPUs. Fixing that, without a managed transcoder and without buying a rack of NVIDIA cards, is the next chapter.