Contents
The last stretch of work made the platform quick — a heavy course page that used to take thirty seconds now answered in about two hundred milliseconds, and the API felt solid enough to build real features on. So we finally turned to the thing we had been deliberately ignoring since the very early days: video. Back when we were still proving people would pay, I had left a landmine in the product on purpose. A single 20-minute course lesson was a 1.2 GB HD file, and we served that same giant file to everyone.
That was our number-one support complaint. Not a crash, not a failed payment — your video ate my internet. Our users watch on phones in small-town India, often on prepaid data packs of a couple hundred rupees a month, and one HD lesson could swallow a day’s worth of data in twenty minutes. The platform had proven people would pay for courses. Now it had to prove they could actually watch them without going broke.
This chapter is how we built video streaming from scratch, why an NGO deliberately chose to build instead of rent it, and the honest bill that owning it put on my desk.
First, what “streaming” even means
If you have never worked with video, the jargon hides a simple idea. Let me build it up.
Start with the naive way, which is what we were doing. You store one big video file. When a user hits play, their phone downloads the file and plays it. The problem is right there in the word download: strictly, the player has to pull down enough of the file to play, and for a single fat file with no cleverness, a weak connection means a spinner while the whole thing crawls in. And every viewer gets the identical 1.2 GB regardless of whether they are on office WiFi or a congested rural tower.
Transcoding is the first fix. To transcode is simply to re-encode a video into different formats, resolutions and quality levels. You take the one fat HD source and produce a ladder of versions of it — a tiny 240p rung for a bad 3G connection, a middle 360p rung, a crisp 720p rung — each a separate file at a separate bitrate. The word for serving these adaptively is adaptive bitrate streaming: the player is allowed to switch between rungs on the fly as the network changes.
HLS (HTTP Live Streaming) is the format that makes that switching work, and it is worth understanding properly because it is the heart of everything here. Instead of one file per rung, HLS chops each rung into small chunks of a few seconds each — a 20-minute lesson becomes a couple hundred little six-second .ts segments per rung. Alongside them sits a manifest (an .m3u8 text file) that is just a playlist listing the chunks and the available rungs. The player reads the manifest, then streams only the chunks it needs, one at a time, and it re-decides the quality for each chunk based on how fast the last few arrived.
This is why the buffer bar behaves the way it does. That grey bar creeping ahead of the playhead is literally the chunks the player has already fetched — nothing more. You can scrub back and forth freely within the loaded chunks because they are already on the device; drag past them and the player just asks the manifest which chunks cover that timestamp and fetches those instead. And because a phone on a weak tower only ever pulls small 240p chunks, it downloads a tiny fraction of the data — HLS slashes both what we store and what we ship.
flowchart TD
S[One 1.2 GB HD source] --> T[Transcode into a ladder]
T --> A[240p tiny rung in 6s chunks]
T --> B[360p mid rung in 6s chunks]
T --> G[720p crisp rung in 6s chunks]
A --> M[Master manifest lists every rung]
B --> M
G --> M
M --> PL[Player picks a rung per chunk by live bandwidth]Two more terms and we have the full vocabulary. Object storage — Amazon S3 is the canonical one — is a place to dump files (objects) and fetch them back by a key, like an infinite hard drive you talk to over HTTP. It is cheap, durable and dumb, which is exactly what you want for a pile of video chunks. And a CDN (content delivery network) — CloudFront is AWS’s — is a fleet of caching servers spread around the world; the first viewer in a region pulls a chunk from storage, the CDN keeps a copy near that region, and the next thousand viewers get it from the nearby cache instead of round-tripping to the origin. The cost of shipping those bytes out to viewers is called egress, and remember that word — it becomes the whole plot later.
Build it, or just rent it?
The honest first instinct of any sane engineer is: do not build this. Managed video platforms exist precisely so you never learn what a .ts segment is. Mux and Vimeo OTT will take your upload, produce the HLS ladder, host it, and stream it worldwide behind their own CDN. They are genuinely excellent, and for most companies most of the time they are the right answer. I want to be clear I was not sneering at them.
But I ran the numbers from a non-profit’s balance sheet, and the shape of the pricing was the problem. Managed video is billed on encoding minutes, on storage, and — the one that matters — on per-GB delivery. That last one has a property that is wonderful for the vendor and quietly lethal for an NGO: your cost scales linearly with your success, forever, with no ceiling. Every new user who watches more lessons is a permanently larger monthly bill. For a venture-funded startup burning to grow, fine — that is just the cost of the growth you are buying. For a foundation funded by donations, “every bit of success costs us more each month and there is no top to it” is not a line item, it is an existential shape.
Owning the pipeline changes the shape of the cost rather than just the size. You pay for raw compute to transcode each video once, cheap object storage to hold the chunks, and CDN egress to deliver them — and each of those three is a separate market you can attack independently, swap vendors on, and negotiate. You trade one clean per-minute invoice for three messier bills you actually control. That control was the second reason too: for our audience the bottom of the ladder mattered more than the top, and I wanted to tune an aggressive, ugly, wonderfully small low rung harder than any managed encoder would let me, and to re-encode the entire catalogue the day I found a better setting.
Stated plainly, the trade I made: more engineering that I now had to keep alive, in exchange for a cost curve an NGO could survive and a bitrate ladder shaped to the exact phones our users held. On those terms it was not close.
The pipeline we actually built
Here is the whole thing, end to end, in the Django rebuild.
flowchart LR
U[Content team uploads source] --> I[Ingest bucket on S3]
I --> Q[Celery transcode task on ECS]
Q --> F[ffmpeg builds the HLS ladder]
F --> D[Delivery bucket on S3]
D --> CF[CloudFront CDN]
CF --> P[Player on a phone in India]Upload. The content team is non-technical, so uploading had to be a file picker, not a command line. They add a source video through the Django admin and it lands in a private S3 ingest bucket, keyed by course and lesson. Nothing is ever served from here — this bucket only holds fat originals.
Job dispatch. The upload fires off a Celery task. Celery is a task queue: instead of doing slow work inside the web request while the user waits, you drop a note on a queue and a separate pool of worker processes picks it up and does it in the background. Transcoding is the textbook case — it takes minutes, not milliseconds. Crucially we gave transcoding its own dedicated queue and its own workers, because an ffmpeg job pegs a CPU core flat for minutes, and you never want that starving the workers handling logins and payments. This is the same isolation principle the whole platform runs on — the payments queue must never wait behind a transcode queue:
# settings.py — one broker, many isolated lanes
CELERY_TASK_ROUTES = {
"transcode.tasks.transcode_to_hls": {"queue": "transcode"},
"payments.tasks.*": {"queue": "payments"},
"notifications.tasks.*": {"queue": "default"},
}
# acks_late so a worker that dies mid-encode redelivers the job instead of losing it;
# prefetch_multiplier=1 so a slow ffmpeg job never hogs a batch of queued work.
CELERY_TASK_ACKS_LATE = True
CELERY_WORKER_PREFETCH_MULTIPLIER = 1
Each lane gets its own worker fleet, sized to its workload. Payments are quick and bursty, so they run wide and cheap; transcodes are long and CPU-bound, so they run narrow with concurrency pinned to the core count:
# payments: many light workers, high concurrency
celery -A app worker -Q payments -c 4 --hostname=pay@%h
# transcode: one heavy worker per core, ffmpeg saturates a core on its own
celery -A app worker -Q transcode -c 1 --hostname=transcode@%h
Those transcode workers run on ECS, AWS’s container service, so we can scale that pool independently of everything else. Sizing matters here: a CPU libx264 encode of a 720p rung is memory-light but core-hungry, so each ECS task was a 1 vCPU / 2 GB Fargate task running a single-concurrency worker — one core, one encode, no oversubscription. Scaling was “run more tasks,” not “make the task bigger,” because ffmpeg does not get faster on a box it cannot use.
Transcode. A worker pulls the original from the ingest bucket, runs ffmpeg to produce the ladder, and pushes the rungs and manifests to a separate delivery bucket. Here is the task, close to what actually shipped:
# transcode/tasks.py
import subprocess
import tempfile
from pathlib import Path
import boto3
from celery import shared_task
from django.conf import settings
from .models import Video
# One rung = one resolution + its bitrate ceiling. Tuned for mid-range Android on prepaid data.
RUNGS = [
{"w": 426, "h": 240, "v": "300k", "maxrate": "330k", "bufsize": "600k", "a": "64k"},
{"w": 640, "h": 360, "v": "700k", "maxrate": "770k", "bufsize": "1400k", "a": "96k"},
{"w": 1280, "h": 720, "v": "2500k", "maxrate": "2750k", "bufsize": "5000k", "a": "128k"},
]
def _build_ffmpeg_cmd(src: Path, out_dir: Path) -> list[str]:
# Split the decoded source into one branch per rung, scale each branch independently.
splits = "".join(f"[v{i}]" for i in range(len(RUNGS)))
filters = [f"[0:v]split={len(RUNGS)}{splits}"]
for i, r in enumerate(RUNGS):
filters.append(f"[v{i}]scale=w={r['w']}:h={r['h']}[v{i}out]")
filter_complex = "; ".join(filters)
cmd = ["ffmpeg", "-y", "-i", str(src), "-filter_complex", filter_complex]
for i, r in enumerate(RUNGS):
cmd += [
"-map", f"[v{i}out]",
f"-c:v:{i}", "libx264",
f"-b:v:{i}", r["v"], f"-maxrate:v:{i}", r["maxrate"], f"-bufsize:v:{i}", r["bufsize"],
"-map", "a:0", f"-c:a:{i}", "aac", f"-b:a:{i}", r["a"],
]
var_map = " ".join(f"v:{i},a:{i}" for i in range(len(RUNGS)))
cmd += [
# Keyframe alignment — the single most important flag group in this whole file.
"-g", "48", "-keyint_min", "48", "-sc_threshold", "0",
"-preset", "veryfast", "-profile:v", "main",
"-f", "hls",
"-hls_time", "6", "-hls_playlist_type", "vod", "-hls_segment_type", "mpegts",
"-hls_segment_filename", str(out_dir / "stream_%v" / "seg_%03d.ts"),
"-master_pl_name", "master.m3u8",
"-var_stream_map", var_map,
str(out_dir / "stream_%v" / "index.m3u8"),
]
return cmd
@shared_task(
bind=True,
queue="transcode",
acks_late=True,
max_retries=2,
autoretry_for=(subprocess.CalledProcessError,),
retry_backoff=30,
)
def transcode_to_hls(self, video_id: int) -> None:
video = Video.objects.select_related("course").get(pk=video_id)
s3 = boto3.client("s3")
with tempfile.TemporaryDirectory() as tmp:
tmp_path = Path(tmp)
src = tmp_path / "source.mp4"
out_dir = tmp_path / "hls"
for i in range(len(RUNGS)):
(out_dir / f"stream_{i}").mkdir(parents=True, exist_ok=True)
# 1. Pull the fat original out of the INGEST bucket.
s3.download_file(settings.INGEST_BUCKET, video.source_key, str(src))
# 2. Encode. check=True turns a non-zero ffmpeg exit into CalledProcessError,
# which the autoretry_for above catches and re-queues.
subprocess.run(_build_ffmpeg_cmd(src, out_dir), check=True, capture_output=True)
# 3. Push every rung + manifest to the DELIVERY bucket under this video's prefix.
for path in out_dir.rglob("*"):
if path.is_file():
key = f"{video.hls_prefix}/{path.relative_to(out_dir)}"
ctype = "application/vnd.apple.mpegurl" if path.suffix == ".m3u8" else "video/mp2t"
s3.upload_file(str(path), settings.DELIVERY_BUCKET, key,
ExtraArgs={"ContentType": ctype})
Video.objects.filter(pk=video_id).update(status=Video.Status.READY)
The ffmpeg invocation it builds is the heart of everything, so read it slowly:
ffmpeg -y -i source.mp4 \
-filter_complex "[0:v]split=3[v0][v1][v2]; \
[v0]scale=w=426:h=240[v0out]; \
[v1]scale=w=640:h=360[v1out]; \
[v2]scale=w=1280:h=720[v2out]" \
-map "[v0out]" -c:v:0 libx264 -b:v:0 300k -maxrate:v:0 330k -bufsize:v:0 600k -map a:0 -c:a:0 aac -b:a:0 64k \
-map "[v1out]" -c:v:1 libx264 -b:v:1 700k -maxrate:v:1 770k -bufsize:v:1 1400k -map a:0 -c:a:1 aac -b:a:1 96k \
-map "[v2out]" -c:v:2 libx264 -b:v:2 2500k -maxrate:v:2 2750k -bufsize:v:2 5000k -map a:0 -c:a:2 aac -b:a:2 128k \
-g 48 -keyint_min 48 -sc_threshold 0 \
-preset veryfast -profile:v main \
-f hls -hls_time 6 -hls_playlist_type vod -hls_segment_type mpegts \
-hls_segment_filename "stream_%v/seg_%03d.ts" \
-master_pl_name master.m3u8 \
-var_stream_map "v:0,a:0 v:1,a:1 v:2,a:2" \
"stream_%v/index.m3u8"
A few things in there are not optional, and getting them wrong produces a ladder that looks fine and stutters on real phones:
libx264, noth264.h264is the codec;libx264is ffmpeg’s actual software encoder for it. Ask for-c:v h264and you get the wrong (or missing) encoder. On the GPU box later this becomesh264_nvenc— same codec, hardware encoder.-g 48 -keyint_min 48 -sc_threshold 0— keyframe alignment, and this is the one people miss. A keyframe (I-frame) is a fully self-contained frame a player can start decoding from cold; the frames between keyframes only describe changes, so you cannot begin playback on one. HLS can only cut a new segment at a keyframe. Now the trap: adaptive streaming works by letting the player swap rungs at a segment boundary — finish a 240p segment, fetch the next one from 720p. That only works if segment N covers the exact same slice of the timeline on every rung. If each rung places keyframes wherever its own encoder feels like it, the segments don’t line up, the player can’t cleanly splice one rung into the next, and you get a visible hitch or an audio pop on every quality change. Forcing a keyframe every 48 frames (-g 48), forbidding early ones (-keyint_min 48), and disabling ffmpeg’s “insert a keyframe when the scene cuts” heuristic (-sc_threshold 0) makes every rung place its keyframes at identical timestamps. At 24 fps, 48 frames is exactly 2 seconds, so a 6-second segment is 3 clean GOPs on every rung — perfectly switchable.-maxrate/-bufsizeper rung.-b:valone is an average target; a complex scene can still spike far above it and blow a phone’s buffer.-maxratecaps the peak and-bufsizesets the encoder’s rate-control window, so the 240p rung genuinely stays tiny even during motion — which is the whole promise we made to someone on a 3G tower.-hls_segment_type mpegts+-hls_time 6. MPEG-TS.tssegments are the maximally compatible container for the ancient Android WebViews our audience runs. Six-second segments are short enough to react to a dropping connection within a chunk or two, long enough that we are not drowning in per-request overhead.-var_stream_map+-master_pl_name. This is what makes it a ladder and not three unrelated videos:-var_stream_map "v:0,a:0 v:1,a:1 v:2,a:2"tells ffmpeg to emit three variant playlists (one per rung, each pairing its video and audio), and-master_pl_name master.m3u8writes the top-level manifest that lists all three so the player can choose between them.
A 1.2 GB source comes out as a few hundred MB across all rungs combined — and, the entire point, a phone that only ever climbs to 360p pulls a small fraction of even that.
Storage. Originals and outputs never share a bucket — different lifecycle, different access, different blast radius if something leaks. The ingest bucket holds fat originals we only need until the encode succeeds, so a lifecycle rule sweeps them to cold storage and then deletes them; there is no reason to pay hot-storage rates for a 1.2 GB source we will never read again:
{
"Rules": [{
"ID": "expire-ingest-sources",
"Status": "Enabled",
"Filter": { "Prefix": "sources/" },
"Transitions": [{ "Days": 7, "StorageClass": "GLACIER" }],
"Expiration": { "Days": 30 }
}]
}
The delivery bucket is the opposite: it must stay private, yet be readable only through CloudFront so nobody can hit the S3 URL directly and bypass the paywall or the cache. Modern CloudFront does this with Origin Access Control (OAC) — the CDN signs its origin requests with SigV4, and the bucket policy trusts only that one distribution:
{
"Version": "2012-10-17",
"Statement": [{
"Sid": "AllowCloudFrontOACOnly",
"Effect": "Allow",
"Principal": { "Service": "cloudfront.amazonaws.com" },
"Action": "s3:GetObject",
"Resource": "arn:aws:s3:::ht-delivery/*",
"Condition": {
"StringEquals": {
"AWS:SourceArn": "arn:aws:cloudfront::123456789012:distribution/EDFDVBD6EXAMPLE"
}
}
}]
}
With BlockPublicAccess on and that policy in place, a direct GET to the S3 object returns 403; the only door into the delivery bucket is the CDN, and the CDN, as we are about to see, only opens for a signed URL.
Signed delivery. Courses are paid, so delivery has to check entitlement — and that check lives in our code, not the CDN’s. The division of labour is the important idea: CloudFront enforces that a URL is signed; Django decides who deserves a signature. That decision is a business rule, and business rules belong where pytest can test them and I can change them without filing a vendor ticket.
The signing itself is real RSA. You upload a public key to a CloudFront key group; you keep the private key in your app (for us, an env-injected secret, never in the repo); and you sign each URL with botocore’s CloudFrontSigner, handing it an rsa_signer closure built on the cryptography library:
# delivery/signing.py
from datetime import datetime, timedelta, timezone
from functools import lru_cache
from botocore.signers import CloudFrontSigner
from cryptography.hazmat.primitives import hashes, serialization
from cryptography.hazmat.primitives.asymmetric import padding
from django.conf import settings
from django.core.exceptions import PermissionDenied
@lru_cache(maxsize=1)
def _signer() -> CloudFrontSigner:
# Private key PEM comes from the environment, not the repo.
private_key = serialization.load_pem_private_key(
settings.CLOUDFRONT_PRIVATE_KEY.encode(),
password=None,
)
def rsa_signer(message: bytes) -> bytes:
# CloudFront canned policies are signed RSA-SHA1 over the policy statement.
return private_key.sign(message, padding.PKCS1v15(), hashes.SHA1())
# The key-pair id ties this signature to the public key sitting in the CloudFront key group.
return CloudFrontSigner(settings.CLOUDFRONT_KEY_PAIR_ID, rsa_signer)
def signed_manifest_url(video, user) -> str:
# Entitlement check FIRST — no signature ever exists for a user who has not paid.
if not user.has_access(video.course):
raise PermissionDenied("Not entitled to this course")
master = f"https://cdn.example.org/{video.hls_prefix}/master.m3u8"
expires = datetime.now(timezone.utc) + timedelta(hours=6) # link dies with the session
# A canned policy authorises exactly ONE URL. HLS is a tree of files, so we
# sign a *custom* policy whose Resource is a wildcard over the whole prefix —
# master.m3u8, every variant playlist, and every .ts segment underneath it.
resource = f"https://cdn.example.org/{video.hls_prefix}/*"
policy = _signer().build_policy(
resource,
date_less_than=expires,
)
return _signer().generate_presigned_url(master, policy=policy)
The wildcard matters more than it looks. My first cut signed a canned policy (date_less_than= with no policy=), which authorises exactly one URL — master.m3u8. The player would load the master fine, then fire off requests for the variant playlists and hundreds of .ts segments under the same prefix, and every one of those came back 403: the signature on the query string was minted for a different resource. CloudFrontSigner.build_policy with a Resource of <prefix>/* fixes it — one signature now covers the entire ladder for its lifetime, so one entitlement check unlocks the whole lesson rather than just its table of contents. (The same policy can instead be handed to CloudFront as three signed cookies — CloudFront-Policy, -Signature, -Key-Pair-Id — which is what you want if you would rather not stamp a long signature onto every segment URL. Either way it is one door, one key, the entire lesson behind it.)
What owning it bought, and what it cost
It worked, and the win was immediate. Videos played, adapted to bad connections, and the “your video ate my data” complaints fell off a cliff — a lesson at 360p is a rounding error against what a 1.2 GB HD download used to cost. We owned the ladder, so when we wanted the bottom rung smaller we changed the encode and re-ran the queue across the catalogue. And the recurring cost was raw infrastructure we could attack, not a per-GB toll gate that grew every single time we succeeded.
And then owning it started mailing me invoices.
The first crack was speed. The transcode workers were fine right up until the content team decided they liked the new pipeline and dropped 150-plus videos in a single day. CPU ffmpeg on ECS fell hopelessly behind — a backlog measured first in hours, then in days — because one worker encoding one video at a time simply cannot keep pace with that. That is the very next problem in this story.
The second crack was scarier and came from delivery. One ordinary day, roughly 5,000 users watched content and CloudFront egress cost us about 1,500 USD in twenty-four hours. For a SaaS that is a shrug. For a non-profit it was a siren. The lesson landed hard: I had traded a per-GB vendor bill I understood for an egress bill that was also per-GB, just wearing an AWS badge. Building had not repealed the laws of physics — bytes still cost money to move — it had only bought me the freedom to go find someone who moved them cheaper.
The lesson: own the pipeline your economics can’t afford to rent
The transferable rule is not “always build video in-house.” For most teams, most of the time, rent it — Mux is worth every rupee and your engineers are worth far more pointed at your actual product. Build it only when the rented cost curve is incompatible with the shape of your business, and here it genuinely was: an NGO whose success would otherwise mean an unbounded, ever-climbing streaming bill, serving an audience for whom the cheap bottom of the bitrate ladder was the feature.
But walk in with both eyes open, because the moment you own the pipeline you own its scaling problems. Renting folds transcoding speed, storage growth, and egress cost into one predictable line item somebody else loses sleep over. Building unbundles them into three separate problems that are now yours to solve, on your own, at 2 a.m. That is a fair trade only if you are actually going to solve them.
We were, so it was. The build was the right call — I would make it again. It just came with a to-do list I had not finished reading, and the item at the very top, the one already blinking red, was that our own transcoding was far too slow to keep up.