Contents
In the last chapter we owned the whole video pipeline: content uploaded to an S3 ingest bucket, a Celery worker on ECS ran ffmpeg to build the adaptive-bitrate ladder, and CloudFront served the HLS chunks. It worked beautifully the day I demoed it with three test videos. Then the content team found the bulk uploader.
I opened Grafana on a Tuesday and the transcode_video queue depth was a straight diagonal line, climbing since nine that morning. Not a spike that drains — a line that only went up. The team had discovered they could select a course’s entire back-catalogue and push it in one sitting, and they were doing exactly what a good content team should do: 150-plus videos before lunch. My CPU workers were going to be chewing on that pile until Friday.
This is the good kind of problem. It means people are using the thing. But a lesson uploaded Tuesday that only becomes watchable Thursday is still a broken promise to the person who paid for the course.
Why transcoding eats CPUs alive
To see why the queue never drained, you have to know what the encoder is actually doing, because “convert the video” hides an enormous amount of arithmetic.
A 20-minute video at 24 frames per second is roughly 28,800 frames. Each frame in HD is roughly two million pixels. The naive way to store that is every pixel of every frame — which is exactly the 1.2 GB monster we started with. Compression avoids it by being clever, and clever is expensive. For most frames the encoder does not store the picture at all; it stores how this frame differs from the one before it. To do that it plays a matching game called motion estimation: it chops the frame into little blocks and, for each block, searches the previous frame for the patch that looks most like it, so it can say “this block is that block from before, shifted eleven pixels to the right.” Then it stores only the shift and the tiny leftover difference.
That search is the cost. Searching harder — a wider window, sub-pixel precision, more candidate reference frames — gives better compression per byte, which is the entire point when your users are on metered mobile data. Then, on top of the search, every block gets a mathematical transform, quantization, and entropy coding. Multiply all of that by 28,800 frames. Then multiply again by every rung of the ladder, because 240p, 360p, 480p, 720p, and 1080p are five separate encodes, not one.
Here is the actual command my ECS worker shelled out to for a single 720p rung. The flags are not decoration — every one of them is load-bearing for adaptive-bitrate HLS, and they are exactly why the CPU sweated.
ffmpeg -i ingest/lesson-8842.mp4 \
-c:v libx264 -preset slow -crf 21 -profile:v high \
-vf scale=-2:720 \
-g 48 -keyint_min 48 -sc_threshold 0 \
-c:a aac -b:a 128k -ac 2 \
-hls_time 6 -hls_playlist_type vod -hls_segment_type mpegts \
-hls_segment_filename '720p/seg_%03d.ts' 720p/index.m3u8
The video codec is libx264 — the software H.264 encoder, not a magic h264 alias (there isn’t one; get the name wrong and ffmpeg just dies). -preset slow is the knob that buys quality-per-byte by searching harder, and it is precisely the knob that costs wall-clock. The three GOP flags are the ones people skip and then wonder why their player stutters on a quality switch: -g 48 -keyint_min 48 -sc_threshold 0 forces a keyframe every 48 frames — two seconds at 24fps — with scene-cut detection disabled, so every rung of the ladder has its keyframes at the exact same timestamps. That alignment is not optional. HLS lets a player jump from the 480p rung to the 720p rung mid-stream when the network improves, but it can only do that at a segment boundary that begins with a keyframe. If the rungs’ keyframes drift, the segments aren’t interchangeable and the switch either fails or glitches. -hls_time 6 cuts six-second segments; -hls_playlist_type vod writes a complete, seekable playlist.
A general-purpose CPU runs all of this in software. It is wonderfully flexible — I can tune every knob — but it is doing millions of tiny searches in sequence, and it is slow. On -preset slow, which we needed for decent quality-per-byte, one 20-minute source took roughly 15 to 20 minutes of wall-clock to produce the full five-rung ladder, and that is with the box doing nothing else.
The arithmetic of the backlog
We ran the transcode_video workers at concurrency 1 on purpose. A transcode pins every core it can grab, so two of them on one host just fight and both go slower. One worker means one video at a time. Three workers, one video each, ~18 minutes per source: that is the whole capacity of the farm.
This is a queue, so it obeys queue rules. The one that matters here is the stability condition: a queue is stable only when the arrival rate λ is strictly less than the total service rate μ. When λ < μ the backlog fluctuates around some finite number and always drains. When λ ≥ μ the backlog has no equilibrium — it grows without bound, forever, for as long as the overload lasts. There is no clever scheduling, no prefetch tweak, no priority trick that saves you once you cross that line, because the work is arriving faster than any arrangement of your fixed servers can retire it.
Put our numbers in. Service rate per worker is one video per ~18 minutes, so across a working day a worker clears roughly 30 videos flat out; three of them, with real-world overhead and the box also doing other things, cleared 30 to 40 a day combined. Call it μ ≈ 40/day. On a quiet day λ might be 10 and the queue is empty by lunch. On a course-launch day λ is 150 to 200.
λ = 175/day, μ = 40/day
net accumulation = λ − μ = +135 videos/day
That plus sign is the whole story. Every busy day adds 135 videos of debt that the quiet days can only chip 30/day off of. One 200-video launch takes the better part of a week to digest even if nobody uploads another thing — and they always upload another thing.
flowchart LR
U[Content team<br/>arrival 175 a day] --> Q[transcode_video queue]
Q --> W1[CPU worker 1<br/>one video at a time]
Q --> W2[CPU worker 2<br/>one video at a time]
Q --> W3[CPU worker 3<br/>one video at a time]
W1 --> B[Service 40 a day total]
W2 --> B
W3 --> B
B --> D[Deficit 135 a day<br/>backlog grows without bound]Inbound: 175 on a launch day. Throughput: 40. This queue did not have a busy hour that passed — it had λ > μ, a permanent, growing deficit. Every course launch pushed the “your video is ready” moment further into next week. The fix cannot be “make μ a bit bigger”; it has to be “make μ elastic enough that λ can never overtake it.”
The conventional move, and why I didn’t make it
The obvious fix is more iron: bigger CPU boxes, or a rack of GPU machines running hardware-accelerated encoding, and just encode faster.
Hardware encoding is worth understanding here, because it is the tempting answer. Instead of doing motion estimation in software, chips like NVIDIA’s NVENC have a dedicated fixed-function block etched into the silicon whose only job is video encoding. It does the matching game in hardware, dozens of times faster than a CPU, at a modest cost in compression efficiency. A GPU box could genuinely chew through the ladder in a fraction of the time.
I have since run exactly that GPU setup, and it earns its own chapter. But at this moment it was the wrong call, for a reason that has nothing to do with encoding speed: the workload is spiky.
The content team does not upload 40 videos every day. They upload 150 on Tuesday, 200 the day a new course lands, and zero for the next week while they record the next batch. If I size a fixed CPU or GPU fleet for the peak, I pay for idle silicon ninety percent of the time — on a non-profit’s budget. If I size it for the average, the queue never drains on a big day, which is the exact pain I am trying to kill. Bursty, bimodal workloads are miserable to provision with fixed capacity: you are always either burning money on idle machines or missing the deadline. And every one of those machines is a box I have to patch, monitor, and wake up for at 2 a.m.
What a managed transcoding service actually is
There is a third option that fits a spiky workload the way fixed hardware never can: rent the farm instead of owning boxes.
A managed transcoding service is a vendor who runs an enormous, elastic fleet of encoders so you never have to. You hand them a source file and a recipe — give me these five rungs, chopped into HLS chunks, written to this bucket — and they spin up as much encoding capacity as the job needs, run it, drop the output in your S3, and bill you per output-minute. Ten videos or two hundred, the wait per video is roughly the same, because behind the API they run your encodes in parallel across their fleet. You pay only for minutes you actually encode, and there is nothing to patch when the queue is empty.
AWS MediaConvert is exactly this, and because our storage and CDN already lived in AWS, it slotted in with no new vendor relationship. The mental shift is the important part: I stop treating transcoding as my compute problem and start treating it as a job I dispatch and a file I collect.
How I actually wired it in
I did not rip out Django and Celery. The pipeline kept its shape — the only change was who does the heavy lifting. The Celery worker used to be the encoder; now it is the orchestrator. It stops running ffmpeg and instead submits a MediaConvert job, then gets out of the way. The worker is free again in milliseconds instead of pinned for twenty minutes.
The endpoint is worth a note: MediaConvert hands each account its own account-specific endpoint URL, which you fetch once (from a regional mediaconvert client’s describe_endpoints) and then cache in settings rather than looking up on every call.
import boto3
# Account-specific endpoint, fetched once via describe_endpoints and cached.
client = boto3.client(
"mediaconvert",
region_name="ap-south-1",
endpoint_url=settings.MEDIACONVERT_ENDPOINT,
)
def submit_transcode(source_key: str, dest_prefix: str, asset_id: int) -> str:
"""Dispatch one HLS ABR ladder job to MediaConvert and return its id."""
response = client.create_job(
Role=settings.MEDIACONVERT_ROLE_ARN,
UserMetadata={"asset_id": str(asset_id)}, # echoed back on completion
Settings={
"Inputs": [{
"FileInput": f"s3://{settings.INGEST_BUCKET}/{source_key}",
"AudioSelectors": {"Audio Selector 1": {"DefaultSelection": "DEFAULT"}},
"TimecodeSource": "ZEROBASED",
}],
"OutputGroups": [{
"Name": "Apple HLS",
"OutputGroupSettings": {
"Type": "HLS_GROUP_SETTINGS",
"HlsGroupSettings": {
"Destination": f"s3://{settings.OUTPUT_BUCKET}/{dest_prefix}",
"SegmentLength": 6,
"MinSegmentLength": 0,
"SegmentControl": "SEGMENTED_FILES", # .ts files, not fMP4
"ManifestDurationFormat": "INTEGER",
"DirectoryStructure": "SINGLE_DIRECTORY",
},
},
"Outputs": [
_h264_rendition(width=426, height=240, bitrate=400_000, name="_240"),
_h264_rendition(width=640, height=360, bitrate=800_000, name="_360"),
_h264_rendition(width=854, height=480, bitrate=1_400_000, name="_480"),
_h264_rendition(width=1280, height=720, bitrate=2_800_000, name="_720"),
_h264_rendition(width=1920, height=1080, bitrate=5_000_000, name="_1080"),
],
}],
},
)
return response["Job"]["Id"]
def _h264_rendition(width: int, height: int, bitrate: int, name: str) -> dict:
"""One rung of the ladder. GOP of 2s, keyframes aligned so rungs are switchable."""
return {
"NameModifier": name, # appended to each segment/manifest name
"ContainerSettings": {"Container": "M3U8", "M3u8Settings": {}},
"VideoDescription": {
"Width": width,
"Height": height,
"CodecSettings": {
"Codec": "H_264",
"H264Settings": {
"RateControlMode": "QVBR", # quality-targeted VBR
"QvbrSettings": {"QvbrQualityLevel": 7},
"MaxBitrate": bitrate,
"GopSize": 48, # 2s at 24fps
"GopSizeUnits": "FRAMES",
"GopClosedCadence": 1,
"SceneChangeDetect": "DISABLED", # keep keyframes aligned across rungs
"CodecProfile": "HIGH",
},
},
},
"AudioDescriptions": [{
"CodecSettings": {
"Codec": "AAC",
"AacSettings": {"Bitrate": 128_000, "CodingMode": "CODING_MODE_2_0", "SampleRate": 48_000},
},
}],
}
Read the _h264_rendition helper next to the ffmpeg command from earlier and it is the same recipe in a different dialect. GopSize: 48 + GopSizeUnits: FRAMES + SceneChangeDetect: DISABLED is MediaConvert’s spelling of -g 48 -keyint_min 48 -sc_threshold 0 — the exact keyframe-alignment discipline that makes the rungs interchangeable, now enforced by the service instead of by me getting the flags right. HLS_GROUP_SETTINGS with SegmentControl: SEGMENTED_FILES produces the same .ts segments and .m3u8 manifests CloudFront was already serving, so nothing downstream noticed the swap. In practice the whole ladder lives in a saved job template and the Celery task just passes the input, the destination, and the asset_id — I’ve inlined the full Settings here so you can see the shape the template captures.
The one genuinely new piece was learning when a job finished. I was not going to poll get_job in a loop for twenty minutes and call that progress — that is a busy-wait holding a worker slot for no reason. MediaConvert emits state changes to EventBridge, so a COMPLETE or ERROR job fires an event I match with a rule and route to a handler.
{
"source": ["aws.mediaconvert"],
"detail-type": ["MediaConvert Job State Change"],
"detail": { "status": ["COMPLETE", "ERROR"] }
}
The event carries back the userMetadata we stuffed in at submit time, so the handler never has to remember which job belonged to which lesson — the correlation rides along in the event.
def handle_mediaconvert_event(event: dict) -> None:
"""Invoked by EventBridge on job COMPLETE or ERROR. No polling."""
detail = event["detail"]
asset_id = int(detail["userMetadata"]["asset_id"])
status = detail["status"]
if status == "COMPLETE":
LessonAsset.objects.filter(pk=asset_id).update(
status=LessonAsset.Status.READY,
hls_master_manifest=detail["outputGroupDetails"][0]["playlistFilePaths"][0],
)
else: # ERROR
LessonAsset.objects.filter(pk=asset_id).update(
status=LessonAsset.Status.FAILED,
failure_reason=detail.get("errorMessage", "unknown"),
)
# page a human; a failed encode is a lesson nobody can watch
notify_content_ops(asset_id, detail.get("errorMessage"))
Submit, walk away, get told when it is done — or told loudly when it broke. Push, not poll.
flowchart LR
U[Upload to S3 ingest] --> C[Celery worker submits job]
C --> M[MediaConvert<br/>elastic encode fleet]
M --> S[HLS ladder to S3 output]
M --> E[EventBridge job complete]
E --> H[Handler marks lesson ready]
S --> CF[CloudFront to viewers]Two things I made sure of before cutting over. First, the ingest and output buckets stayed exactly as the in-house pipeline left them, so the player, the manifests, and CloudFront never knew anything had changed — MediaConvert was a drop-in replacement for the ffmpeg step, nothing more. Second, I kept the old ffmpeg task in the codebase behind a flag, so if MediaConvert produced a ladder I did not like, I could fall back without a deploy.
The outcome
The backlog cleared. The next time the team dumped a 150-video course in a morning, the queue depth on my dashboard spiked and then fell back to zero within the hour, because those 150 jobs ran in parallel on someone else’s fleet instead of single-file through three of my workers. A lesson uploaded at 10 a.m. was watchable before lunch. The “your video is ready” delay stopped being measured in days and started being measured in minutes.
My Celery workers went back to being web-app workers. They were no longer a video farm impersonating a task queue, holding a CPU hostage for twenty minutes at a stretch. The transcode queue that used to be a source of anxiety became a queue that dispatches jobs and forgets them — exactly what a queue should be.
Fixed fleet versus elastic fleet, in money
The reason this works is that MediaConvert changed the shape of the cost curve, not just the speed. Go back to the stability condition: a fixed fleet gives you a constant μ, so you are choosing a horizontal line and praying λ stays under it. Provision three ECS workers and μ is 40/day whether the team uploads 0 videos or 200 — you pay for the same three boxes both days, and on the 200 day you miss the deadline anyway. To make a fixed fleet never fall behind, you have to size it for the worst launch day you can imagine and then eat that bill 365 days a year, most of them idle.
Managed encoding deletes the horizontal line. There is no μ to under-provision because the service parallelizes each batch across its own fleet — 150 jobs submitted at 10 a.m. run essentially at once, so wall-clock-to-ready barely moves whether the day’s λ is 10 or 200. You stop buying capacity and start buying output-minutes: you pay per minute of video actually encoded, summed across all five rungs, and nothing when the queue is empty.
That is the trade, stated honestly. On a flat, predictable workload a fully-utilized owned box is cheaper per minute than a managed service’s per-minute rate — you are paying someone else’s margin. But our workload was the opposite of flat: bimodal, spiky, idle most of the week. For that shape, per-minute billing that follows the spikes beats a fixed fleet sized for the peak, because you are no longer paying for the ninety percent of the week the peak isn’t happening. The elastic fleet costs money only in proportion to work done; the owned fleet costs money in proportion to work you might someday do.
flowchart LR
A[Fixed ECS fleet<br/>constant service rate] --> B[Idle most days<br/>misses launch spikes]
C[MediaConvert<br/>elastic per minute] --> D[Follows the spike<br/>pay only for minutes encoded]
B --> E[Pay for peak all year]
D --> F[Pay for work actually done]The lesson
Buy managed for the bottleneck that is not your differentiator. My differentiator was a spiritual-education catalogue tuned for cheap Indian mobile data, not a hand-built encoding farm. Video encoding is a solved, commoditized problem that thousands of engineers at a cloud vendor optimize full-time; me babysitting ffmpeg presets on ECS added nothing a user could feel, and it added a fixed-capacity fleet I had to size, patch, and stare at on a Tuesday morning. The moment a spiky, bimodal workload starts dictating your architecture, the right question is not “how do I run this faster” but “should I be running this at all.” Your app’s workers should serve your app. Let someone whose entire business is elastic compute absorb the spikes.
There was, of course, a catch. I had swapped a capacity problem for a pricing problem, and MediaConvert bills per output-minute across all five rungs of every video the team could upload. It cleared the backlog on the first try — and then, at the end of the month, the bill arrived.