Kuldeep Pisda
The 3.5 TB Reckoning: Our Cheapest Setup Was Our Biggest Risk

The 3.5 TB Reckoning: Our Cheapest Setup Was Our Biggest Risk

Our scrappy Wasabi and Cloudflare setup saved money until 3.5 TB and 30,000 users outgrew its fair-use terms. Here is why we moved to Bunny.net for media.

20 min read

The 3.5 TB Reckoning: Our Cheapest Setup Was Our Biggest Risk
Contents

For about eighteen months, the cheapest line on our infrastructure bill was also the most dangerous one. Our media stack — Wasabi for object storage, Cloudflare in front of it as the CDN, and a 600-dollar Ubuntu desktop with an NVIDIA card quietly transcoding video in the corner of an office — was saving us thousands of dollars a month over the AWS setup it replaced. It was also, at roughly 3.5 TB of media and 30,000 users, quietly becoming the wrong shape for every vendor whose plan it leaned on.

This is the chapter where the hack grew up. Not because it broke — it never broke — but because I could finally see that continuing to run it was a bet I no longer wanted to be making with 30,000 people’s access to their courses.

How we ended up with a stack held together by fair use

A quick recap, because the setup is the whole reason the reckoning happened. The 1,500-dollar CloudFront day taught me that egress — the bandwidth you pay for when data leaves a provider’s network toward a user — not compute, was the thing that would bankrupt a non-profit serving video. So we ran from it. We moved storage to Wasabi, whose pitch is near-zero egress fees. We moved the CDN to Cloudflare. And because Wasabi is S3-compatible rather than S3 — which silently broke AWS MediaConvert, since MediaConvert only writes to true S3 — we went back to ffmpeg, prototyped GPU transcoding on vast.ai, decided it wasn’t production-reliable yet, and instead bought an ordinary GPU desktop and wrote a small FastAPI service that pulls transcode jobs, runs ffmpeg on the GPU, and pushes the renditions to Wasabi.

The transcoder itself was boringly correct, which is why it survived so long. It pulled a job, ran a single NVENC pass that produced a keyframe-aligned HLS ladder, and shipped the whole rendition tree to Wasabi. The keyframe alignment is the part people get wrong: every rendition has to cut its GOP at the same wall-clock instants or the player can’t switch bitrates mid-stream without a visible stall, so the GOP is pinned rather than left to the scene-change detector.

# One NVENC pass, three renditions, keyframes locked across all of them.
# -g 48 -keyint_min 48 -sc_threshold 0  => a keyframe every 2s at 24fps,
# identical in every ladder rung, so -hls_time 6 yields switchable segments.
ffmpeg -hwaccel cuda -i "$SRC" \
  -filter_complex "[0:v]split=3[v1][v2][v3]; \
    [v1]scale_cuda=-2:1080[v1o];[v2]scale_cuda=-2:720[v2o];[v3]scale_cuda=-2:480[v3o]" \
  -map "[v1o]" -c:v:0 h264_nvenc -preset p4 -tune hq -rc vbr -cq 23 -b:v:0 0 \
  -map "[v2o]" -c:v:1 h264_nvenc -preset p4 -tune hq -rc vbr -cq 23 -b:v:1 0 \
  -map "[v3o]" -c:v:2 h264_nvenc -preset p4 -tune hq -rc vbr -cq 23 -b:v:2 0 \
  -g 48 -keyint_min 48 -sc_threshold 0 \
  -map a:0 -map a:0 -map a:0 -c:a aac -b:a 128k -ac 2 \
  -f hls -hls_time 6 -hls_playlist_type vod -hls_segment_type mpegts \
  -master_pl_name master.m3u8 \
  -var_stream_map "v:0,a:0 v:1,a:1 v:2,a:2" \
  "$OUT/stream_%v/index.m3u8"

It was scrappy and it was good. It saved a genuinely meaningful amount of money for an NGO, it eventually cached around 600 GB of transcoded content on the desktop’s local disk, and it bought us another long stretch of runway. I would make every one of those calls again at that scale.

But the numbers kept moving. Users grew from a few thousand to about 30,000, roughly 60 percent of them paying. Originals plus transcoded renditions grew to about 3.5 TB. And somewhere along that climb, the workload crossed a line that no dashboard was drawing for me.

Why 3.5 TB was the wall — and it was a hardware wall before it was a policy wall

The fair-use exposure I’ll get to below was the reason we moved. But the reason we had to — the deadline the physics set for us — was the desktop. The GPU box wasn’t just a transcoder; it had quietly become a caching origin. When a rendition was requested and Cloudflare didn’t have it, the pull went back through that desktop, which served it from its local 600 GB cache if it had it and re-fetched from Wasabi if it didn’t. That arrangement has a very specific breaking point, and 3.5 TB is roughly where we hit it.

The cache stopped being a cache. A cache only earns its keep while the hot set fits inside it. At a few thousand users the actively-watched catalogue was a few hundred gigabytes, comfortably inside 600 GB, and the local hit ratio sat in the 90s. As the catalogue crossed 3.5 TB the working set — the renditions being pulled in any given day as courses launched and cohorts moved through them — grew past what 600 GB could hold. The cache started thrashing: every miss evicted something that was about to be needed again, so misses caused more misses.

# desktop cache telemetry, weekly rollup (locally scraped, not a fancy dashboard)
week        catalogue   cache_used   working_set*  hit_ratio   miss->wasabi_GB/day
2024-W31      1.9 TB      591 GB        410 GB        0.94            38
2024-W38      2.6 TB      598 GB        560 GB        0.91            71
2024-W44      3.1 TB      600 GB        690 GB        0.83           186
2024-W49      3.5 TB      600 GB        910 GB        0.71           402   <-- thrash
# *working_set = distinct rendition bytes pulled at least once in the trailing 24h

Watch the last column, not the first. Storage growing is fine; storage is cheap. The problem is that once the working set exceeds the cache, the miss traffic doesn’t grow linearly — it grows super-linearly, because evictions start throwing away bytes that were about to be re-requested. Every one of those misses became a re-fetch from Wasabi to the desktop and then out to the user, which is where the second wall lived.

Home upload bandwidth is a hard ceiling. The desktop lived on an office connection with an asymmetric plan — fat download, thin upload — because that’s what consumer and small-office fibre is. Every cache miss served to a viewer had to leave that building through the upload pipe, and video does not fit through a thin pipe.

def upload_saturation(
    concurrent_streams: int,
    kbps_per_stream: int,       # 720p HLS ~ 2.2 Mbps sustained
    upload_mbps: int,           # office plan uplink
) -> float:
    demand_mbps = (concurrent_streams * kbps_per_stream) / 1000
    return round(demand_mbps / upload_mbps, 2)  # >1.0 means the pipe is the bottleneck

# 40 Mbps uplink, 720p at ~2.2 Mbps, evening prime-time concurrency:
upload_saturation(60, 2200, 40)    # -> 3.30  (need 132 Mbps, have 40)

At a 94 percent hit ratio, only the 6 percent of misses touched that pipe and it never noticed. At 71 percent, nearly a third of every evening’s streams were trying to squeeze back out through a 40 Mbps uplink that physically could not carry them. Buffering on mid-range Android phones on prepaid data — our entire audience — is the first thing that breaks, and it breaks silently, as a support ticket that says “video is slow,” not as a page on my phone.

And it was one desktop. No redundancy, no failover, one power supply, one consumer SSD accumulating writes, sitting in an office that has power cuts. For an MVP that single point of failure is a fine trade. For the delivery origin of 30,000 people’s paid courses it is a liability I had personally signed us up for and quietly stopped being comfortable with. The 3.5 TB number wasn’t a billing threshold. It was the point where a home-grade box was structurally the wrong thing to have load-bearing under a paying platform.

Crash course: what a fair-use policy actually is

Almost every “unlimited” or “free egress” plan in infrastructure carries a quiet clause called a fair-use policy (or acceptable-use policy). It is not a hard cap with a number you can watch approach. It’s a shape the vendor expects your traffic to have — and if your traffic stops looking like that shape, you have drifted out of the deal even though no single meter ever tripped. The bill stays low. The exposure does not. That gap between “no error fired” and “I’m outside the intended use” is exactly where we were living, and understanding it is understanding why we had to move.

Two of these clauses were load-bearing for us. I’ll make both concrete.

The metric nobody puts on a wall chart: egress-to-storage ratio

Here is the thing about Wasabi’s near-zero-egress deal that you only feel at scale. It isn’t unlimited, and it was never meant to be. The fair-use expectation behind it is simple and completely reasonable: your monthly egress should stay roughly at or below the volume of data you actually store. That is a fantastic deal for the workload it was designed for — backups, archives, data lakes, cold storage. You keep a lot, you read a little.

Streaming media is the exact inverse of that workload.

We stored about 3.5 TB. But 30,000 users watching transcoded lessons pull that same data over and over. Even after transcoding cut a view down to roughly 225 MB, the monthly arithmetic is brutal:

def egress_to_storage_ratio(
    stored_tb: float,
    monthly_views: int,
    gb_per_view: float,
) -> float:
    monthly_egress_tb = (monthly_views * gb_per_view) / 1024
    return round(monthly_egress_tb / stored_tb, 1)

# ~120,000 monthly views, ~225 MB each, against 3.5 TB stored:
egress_to_storage_ratio(3.5, 120_000, 0.225)   # -> 7.5

We were comfortably past a 7-to-1 egress-to-storage ratio, and it climbed every single month — against a store whose entire economic model assumes that number stays near 1-to-1. We weren’t abusing Wasabi. We were simply not the customer their pricing was built to serve — a distinction that matters enormously, because one of those is a moral failing and the other is just an architect using a tool outside its design envelope and eventually noticing.

Cloudflare was the same story in a different key. A general-purpose CDN is a spectacular piece of engineering for HTML, JS, CSS, API responses, and images — and its self-serve terms have always, sensibly, steered bulk video and large-file serving toward the products built for exactly that. Caching hundreds of gigabytes of video through a pull zone on a general plan works beautifully right up until you are doing it at a volume those terms were written to exclude. It worked for us because we were small. At 3.5 TB and 30,000 users, “it works because we’re small” had quietly expired.

The honest framing: we outgrew our own gray area

It would be easy to write this as a confession — we were cutting corners and a vendor caught us. That’s not what happened, and reaching for that framing would be the wrong lesson.

What actually happened is more useful. We had built a clever arrangement that made complete sense at the scale where we built it, and then we grew past that scale. Every hack has a domain of validity — a range of load, data volume, and criticality where it is genuinely the right call. Ours had one. The Wasabi-plus-Cloudflare-plus-desktop-GPU stack was the correct engineering for a scrappy platform still proving people would pay. We had simply walked out the far side of that domain without anyone sending us a notification.

Maturity, here, wasn’t “never build the hack.” The hack was right. Maturity was recognizing the day it stopped being clever and started being exposure — and choosing to leave the gray area on our own terms rather than waiting for a vendor’s fair-use email to make the decision for us, on their timeline, with 30,000 users mid-course.

The tell: cost falling, risk rising

If there’s one signal I want other engineering leaders to internalize from this, it’s this pair of curves. Our cost on this stack was low and stayed low. Our risk on this stack rose steadily with every new user, because every new user pushed the egress-to-storage ratio further past the point the free plans were built around. When the thing keeping your bill down is also the thing that could take you offline, that crossing is the signal to act — and it almost never shows up as an alert.

Concretely, the risk had three faces:

  • A single policy email could throttle or cut streaming for 30,000 people in the middle of their courses, with no notice period I controlled.
  • No SLA matched the platform’s new criticality. When 60 percent of your users are paying and the content is load-bearing to the mission, “it’s been fine so far” is not an uptime strategy — it’s a survivorship bias you’re one bad day from disproving.
  • The economics depended on staying in a gray area — an interpretation of two vendors’ fair-use terms — which is a fragile foundation to keep pouring growth onto.

For an MVP, every one of those risks is acceptable; you take them gladly in exchange for speed and cheapness. For proven, load-bearing infrastructure, betting your uptime on a favorable reading of a fair-use clause is not a bet a responsible owner keeps renewing.

Crash course: a purpose-built media platform

The alternative to a general-purpose stack used at its edges is a purpose-built media platform — object storage and a CDN designed, priced, and operated on the assumption that serving lots of video at volume is the product, not an abuse of it. Bunny.net is the one we chose: a Storage Zone (object storage you can geo-replicate to the regions your audience actually lives in) paired with a Pull Zone (its CDN — edge PoPs that read from an origin, cache what they fetch, and serve it near the user). The reasoning is almost embarrassingly simple: with Bunny, heavy egress isn’t tolerated as an exception, it’s the thing you are paying them to do.

That single fact fixes everything that was fragile:

  • The ratio that broke Wasabi’s model is Bunny’s home turf. Bunny prices storage per GB and egress per GB, published and predictable, on the explicit assumption that you will egress far more than you store. A 7-to-1 ratio isn’t a red flag there; it’s Tuesday.
  • No gray area, so no policy risk. There is nothing to outgrow because heavy media delivery is the intended use. The uptime bet stops being an interpretation and becomes a contract.
  • Region-aware pricing that’s sane for India. Serving our actual audience wasn’t punitively priced the way expensive edge regions had been on the old setup.
  • Storage zones and pull zones map cleanly onto the origins-versus-delivery split we already had, so the architecture translated almost one-to-one — as the diagram’s two parallel paths show, only the storage and CDN legs changed; the transcoder and the players didn’t move.

Bunny Stream vs. Storage + CDN — and why we didn’t take the easy button

Bunny sells two ways to do this, and the difference decided our architecture. Bunny Stream is the managed, all-in-one product: you upload a source file and Bunny transcodes it into an HLS ladder, stores it, serves it through its own CDN, and even hands you a player and a per-video GUID. It is genuinely excellent, and for a team without a pipeline it’s the right answer. Storage + CDN is the primitive layer: you bring your own already-transcoded files, drop them in a Storage Zone, and point a Pull Zone at it.

We took Storage + CDN, deliberately, for one reason: we already own the transcoder. A ~$600 desktop with an RTX 4050 running NVENC turns a raw upload into a keyframe-aligned HLS ladder for the cost of electricity. Bunny Stream would have charged us per transcoded GB to redo work our own hardware already does essentially for free, and it would have retired a pipeline that took three vendors’ worth of pain to get right. Stream solves the problem we’d already solved. What we actually needed from Bunny was the one thing our stack didn’t have — a delivery layer built to egress at volume without a fair-use asterisk. So we bought exactly that layer and nothing more. The GPU box keeps transcoding; it just pushes to a Bunny Storage Zone now instead of a Wasabi bucket, and a Pull Zone reads from it.

The pricing, concretely

Here’s the comparison that made the decision unarguable — the same ~27 TB of monthly egress priced three ways. Bunny’s Volume CDN tier is a flat $0.005/GB globally (a slightly smaller PoP network than the Standard tier, which is the honest trade for the price, and still plenty of Asian PoPs for our audience). Storage is $0.01/GB/month per region on the HDD tier; we keep a Singapore main region plus one replica for India latency and durability, so storage is billed twice.

MONTHLY_EGRESS_GB = 27_000        # ~120k views * 225 MB
STORED_GB         = 3_500

def bunny_monthly() -> dict:
    storage = STORED_GB * 0.01 * 2         # HDD $0.01/GB/mo, main + 1 replica
    cdn     = MONTHLY_EGRESS_GB * 0.005    # Volume tier, flat global
    return {"storage": round(storage), "cdn": round(cdn), "total": round(storage + cdn)}

def cloudfront_monthly() -> dict:          # what the spike day extrapolates to
    egress  = MONTHLY_EGRESS_GB * 0.109    # CloudFront India egress, per GB
    storage = STORED_GB * 0.023            # S3 Standard, per GB/mo
    return {"storage": round(storage), "cdn": round(egress), "total": round(egress + storage)}

def wasabi_monthly() -> dict:              # cheap, but the number that could vanish
    storage = STORED_GB * 0.00699          # $6.99/TB/mo
    return {"storage": round(storage), "cdn": 0, "total": round(storage)}

bunny_monthly()       # {'storage': 70,  'cdn': 135,  'total': 205}
cloudfront_monthly()  # {'storage': 80,  'cdn': 2943, 'total': 3024}
wasabi_monthly()      # {'storage': 24,  'cdn': 0,    'total': 24}

Read those three totals together and the call writes itself. Bunny at ~$205/month, fully predictable is roughly 15× cheaper than CloudFront would be at this egress, and only about $180/month more than Wasabi — except Wasabi’s $24 was the number living inside a fair-use gray area that a single policy email could convert into a throttle. We weren’t paying $180/month for bandwidth. We were paying $180/month to delete a category of risk: to turn “our uptime depends on a favorable reading of someone’s AUP” into “our uptime is a line item on an invoice.” For a load-bearing platform, that’s the cheapest insurance on the whole bill.

Wiring it up: push to a Storage Zone, serve from a Pull Zone

Bunny Storage is not S3-compatible — no boto3, no endpoint_url, none of the django-storages plumbing we used for Wasabi. It’s a plain HTTP storage API: PUT to upload, keyed by an AccessKey header that is the Storage Zone’s password. That’s actually a feature here — one fewer SDK, and the upload path is trivial to reason about. The transcoder’s push step changed from a boto3 upload_file to this:

import os
import requests

# Regional storage host: Singapore edge for our audience. DE (default) is
# storage.bunnycdn.com; each region has its own host.
STORAGE_HOST = "https://sg.storage.bunnycdn.com"
STORAGE_ZONE = "happy-thoughts-media"
ACCESS_KEY   = os.environ["BUNNY_STORAGE_ACCESS_KEY"]

def push_rendition(local_dir: str, remote_prefix: str) -> None:
    """Upload an HLS rendition tree (master.m3u8 + variant playlists + .ts) to Bunny Storage."""
    session = requests.Session()
    session.headers.update({"AccessKey": ACCESS_KEY})
    for root, _dirs, files in os.walk(local_dir):
        for name in files:
            path = os.path.join(root, name)
            rel  = os.path.relpath(path, local_dir)
            url  = f"{STORAGE_HOST}/{STORAGE_ZONE}/{remote_prefix}/{rel}"
            with open(path, "rb") as fh:
                resp = session.put(url, data=fh, timeout=60)
            resp.raise_for_status()   # 201 Created on success

Delivery is where the Pull Zone earns its name. You create a Pull Zone whose origin type is the Storage Zone — so the CDN reads directly from Bunny Storage over Bunny’s own backbone, with no cross-provider egress in the middle, the exact leak that made the old Cloudflare-in-front-of-Wasabi hop feel wrong. Bunny gives the zone a default hostname like happy-thoughts.b-cdn.net; you attach your own hostname and let Bunny issue the TLS cert. The HLS just works: the player fetches master.m3u8, the CDN pulls it (and each .ts segment) from the Storage Zone on first request, caches at the edge, and every subsequent viewer in that region is served from cache.

# The whole player-facing surface is three request shapes, all edge-cached:
curl -sI https://cdn.example.org/lessons/anahata-01/master.m3u8   | grep -i 'cdn-cache\|content-type'
#   content-type: application/vnd.apple.mpegurl
#   cdn-cache: HIT                      <- served from the nearest Bunny PoP

curl -s  https://cdn.example.org/lessons/anahata-01/master.m3u8
#   #EXTM3U
#   #EXT-X-STREAM-INF:BANDWIDTH=5000000,RESOLUTION=1920x1080
#   stream_0/index.m3u8
#   #EXT-X-STREAM-INF:BANDWIDTH=2400000,RESOLUTION=1280x720
#   stream_1/index.m3u8
#   #EXT-X-STREAM-INF:BANDWIDTH=1200000,RESOLUTION=854x480
#   stream_2/index.m3u8

The cutover was a DNS change

Because nothing about the player changed — same relative HLS paths, same master.m3u8 layout — the actual switch from Cloudflare-over-Wasabi to a Bunny Pull Zone was a hostname move. The video CDN lived at a stable hostname the apps already pointed at; migrating meant re-pointing that hostname’s CNAME at the Bunny Pull Zone once the catalogue was verified in place. Lower the TTL a day ahead so the flip propagates fast (and so rollback is fast), then swap the target:

; before — video hostname fronted by Cloudflare
cdn.example.org.   300   IN   CNAME   video.cloudflare-proxied.example.net.

; after — same hostname, now the Bunny Pull Zone
cdn.example.org.   300   IN   CNAME   happy-thoughts.b-cdn.net.

One record. Old players resolve to the Bunny edge within the TTL, request the identical paths, and the Pull Zone serves them. If anything had looked wrong in the first minutes, flipping the CNAME back was the entire rollback plan. (The hard part isn’t this line — it’s guaranteeing all 3.5 TB is actually present and byte-correct in the Storage Zone before you flip it. That’s the next chapter.)

The trade is honest and worth stating plainly: Bunny egress is not free the way Wasabi’s technically was. We went from “near-zero egress but living inside someone else’s fair-use gray area” to “a real, per-GB egress price that is predictable, purpose-built, and unambiguously ours to use as hard as we like.” At 30,000 users and climbing, a predictable bill I can forecast a year out beats a near-zero bill that could evaporate into a throttling email overnight. Predictable-and-purpose-built beat clever-and-free — not because clever-and-free was wrong, but because we’d outgrown the range where it was right.

The decision, end to end

The migration deserves its own respect

Moving 3.5 TB of live media between object stores while users stream the entire time — first S3 to Wasabi, now Wasabi to Bunny — is its own discipline, and it’s the subject of the next chapter. The short version: you don’t get a maintenance window. Devotional learners watch at every hour; there is no 3 a.m. when nobody’s mid-lesson. So you keep both buckets at parity — dual-writing every new transcode to old and new while a background job backfills the existing catalogue and verifies it object by object — and only then flip the CDN origin in a single atomic cutover, with rollback ready. Correctness under continuous read traffic is the whole game.

The lesson

Recognizing that you’ve outgrown your own hack is a leadership skill, and it is the opposite of an admission of failure. The scrappy stack was a good decision that saved a real non-profit real money at exactly the moment it needed saving. Building it was senior judgment. Retiring it on time was the same judgment, one turn of the wheel later.

The transferable rule: clever-and-free is the right tool while you’re validating; predictable-and-purpose-built is the right tool once the thing is load-bearing. The maturity isn’t in never reaching for the clever hack — reach for it, it’s often correct. The maturity is in watching your cost curve and your risk curve together, and having the discipline to retire the hack the day they cross, before a vendor’s automated policy enforcement retires it for you at the worst possible moment.

The 1,500-dollar day taught me to stop being surprised by what a system costs when it works. This chapter was the same lesson, aimed one level up: stop being surprised by what a system risks when it works. The cheapest thing on your bill is worth a hard second look — because sometimes the reason it’s cheap is that you’re standing somewhere you’ve quietly outgrown. And the moment you decide to move those 3.5 terabytes to their new home, you inherit a harder question: how do you carry every one of them across without a single viewer’s stream so much as stutter?