Contents
The free-egress move had rescued the NGO’s unit economics and, in the same stroke, handed me back a problem I thought I’d permanently outsourced: transcoding. Wasabi wasn’t real AWS S3, so AWS MediaConvert couldn’t read from it, which meant I was back to encoding video myself with plain ffmpeg — on CPU boxes that could clear maybe 30 to 40 videos a day against an inbound firehose of 150-plus. I needed GPU-speed encoding. I owned zero GPUs. And I worked for an organization that had just spent a week panicking about a four-figure CDN bill, so “let me buy a rack of NVIDIA cards” was not a sentence I could say out loud in a budget meeting.
So I started where a broke engineer with a hard problem should start: at the frontier, renting other people’s idle gaming GPUs by the minute.
Why a GPU eats a CPU’s lunch at video
First, the piece of hardware knowledge that makes this whole chapter make sense, because until it clicked for me I didn’t understand why everyone insists on GPUs for video.
A quick refresher on the job itself: transcoding means taking one big source video and re-encoding it into several smaller, streamable versions — an adaptive-bitrate “ladder” of resolutions, so a phone on a weak 4G pack can pull the 240p rung while someone on office WiFi gets 1080p. Encoding is the expensive step: for every frame, the encoder hunts for how this frame differs from the last one, computes motion vectors, quantizes, and packs the result into H.264 or HEVC. It is a staggering amount of arithmetic per second of footage.
There are two fundamentally different ways to do that arithmetic, and this is the crux:
Software encoding (the x264 library our CPU boxes were running) does the math in general-purpose CPU instructions. The CPU is a brilliant generalist — it can run a database, a web server, or a video encoder — and that flexibility is exactly why it’s slow at any one of them. Software x264 on a good quality preset runs at roughly real-time: a 20-minute clip takes on the order of 20 minutes per rung of the ladder. Multiply by five rungs and 150 videos and you have the multi-day backlog that drove me to a managed service in the first place.
Hardware encoding does the same job in a fixed-function circuit — a block of silicon etched into the chip whose only ability is to run the H.264/HEVC encode pipeline. It can’t run a database. It can’t run anything else. It does one thing, in dedicated transistors, dozens of times faster than the general-purpose path and at a fraction of the power. On NVIDIA GPUs that block is called NVENC (NVIDIA Encoder), and it sits on the die separate from the CUDA cores everyone associates with GPUs.
The mental model that finally made it stick for me: software encoding is a chef cooking every dish to order — infinitely versatile, one plate at a time. NVENC is a factory line stamped to produce exactly one product. Useless for anything else; absurdly fast at the one thing. For a pipeline whose entire job is “same encode, over and over, 150 times a day,” you want the factory line.
The difference isn’t subtle in the command line, either. Same source, same target quality, two encoders. The CPU path leans on the libx264 library (the H.264 software encoder — note it’s libx264, never h264, which is a decoder name people mistype into a broken command):
# CPU / software: libx264. Roughly real-time on a good box — a 20-min
# clip takes ~20 min of wall-clock per rung, and it pins every core.
ffmpeg -i source.mp4 \
-c:v libx264 -preset medium -crf 20 \
-c:a aac -b:a 128k \
out_1080p.mp4
The GPU path hands the whole job to the NVENC block. -hwaccel cuda decodes on the card, -c:v h264_nvenc selects the hardware encoder, and the NVENC-specific rate control (-rc vbr -cq 23 -b:v 0 = constant-quality VBR, no bitrate cap) replaces libx264’s CRF knob:
# GPU / hardware: h264_nvenc. On an RTX-class card a single 1080p rung
# ran at ~8-10x real-time — that 20-min clip encoded in ~2 min per rung.
ffmpeg -hwaccel cuda -i source.mp4 \
-c:v h264_nvenc -preset p4 -tune hq -rc vbr -cq 23 -b:v 0 \
-c:a aac -b:a 128k \
out_1080p.mp4
The wall-clock gap was the entire pitch: libx264 at roughly 1x real-time versus NVENC at eight-to-ten times real-time on a single rung, at a fraction of the power draw and freeing the CPU entirely. Multiply that across five rungs and 150 videos a day and the multi-day backlog collapses into a rounding error.
That reframed my shopping list. I didn’t need compute. I needed access to an NVENC block. The question was how to rent one without owning it.
The marketplace GPU: an Airbnb for graphics cards
The obvious answer is the big clouds — AWS, GCP, Azure all rent GPU instances. But there’s a scrappier corner of the market, and it’s worth understanding because it’s genuinely a different economic model.
A GPU marketplace like vast.ai is closer to Airbnb or Uber than to AWS. Instead of one company operating datacenters, thousands of individuals and small hosts list their idle GPUs — gaming rigs that sit dark while their owner is at work, ex-crypto-mining machines, homelab boxes — and you rent them by the minute at whatever the going spot price is. Because you’re renting someone’s spare capacity rather than reserved datacenter iron, it’s cheap: a card that costs a couple of dollars an hour on-demand in a cloud can go for a fraction of that on the marketplace.
There’s a catch baked into the model, and it’s the whole story of this chapter, so I’ll name it now. Marketplace and spot capacity is interruptible. In cloud terms, a “spot” or “preemptible” instance is one you rent at a steep discount on the explicit condition that the provider — or, on a marketplace, the machine’s actual owner — can reclaim it with little or no warning. The gamer sits down to play; your rented GPU vanishes. You traded reliability for price, whether or not you fully appreciated the trade at signup.
The options I weighed
I had two credible ways to get NVENC cycles without buying hardware, and one non-starter.
- Non-starter: buy GPUs up front. Right idea, wrong month. After the CloudFront scare, capital expenditure on NVIDIA silicon was politically dead. (It comes back in the next chapter, at a price that surprised everyone — but not yet.)
- Option A: first-party cloud GPU instances. Spin up an AWS
g4dnor a GCP T4 box on demand, runffmpegwith NVENC, tear it down. Reliable, in a real datacenter, backed by an SLA. But the sticker price is real — a GPU instance left running is a few hundred dollars a month — and the bursty shape of our load made that worse: 150 videos on a Tuesday, near-silence for days. Sizing a persistent instance for the burst meant paying for a GPU that idled most of the month. Sizing it to spin up per-burst meant eating cold starts and quota approvals, and GPU quota on the big clouds is not always granted to a small NGO account just for asking. - Option B: marketplace GPUs (vast.ai). An order of magnitude cheaper, and pay-per-minute matched our bursty load beautifully — rent a card exactly when a video lands, release it the moment the encode is done. The cost of the interruptibility was, at prototype time, an unknown I wanted to measure rather than assume.
The pragmatic move was to prototype on the cheap frontier first. If vast.ai’s economics held, I’d know the real marginal cost of a transcode and could argue from a number. If the reliability didn’t hold, I’d have learned that cheaply, on a spare weekend, instead of during a production incident.
What I actually built
The architecture was deliberately dumb, and I mean that as praise. The Django backend never talked to a GPU. When a video was uploaded it did one boring thing — wrote a pending row into a transcode-jobs table. A small orchestrator process watched that table and, for each pending job, ran a rent-transcode-return cycle against the marketplace.
flowchart LR U[Course video uploaded] --> DB[(pending job row)] DB --> O[Orchestrator] O -->|rent cheapest offer| G[vast.ai GPU] W[(Wasabi source)] -->|pull original| G G -->|NVENC ffmpeg ladder| G G -->|push renditions| W2[(Wasabi renditions)] G -->|mark done then destroy| DB
The heart of the marketplace model is the lifecycle: find the cheapest suitable card, rent it, push the work, collect the result, and — critically — destroy it immediately so the meter stops. That destroy step is the difference between cents-per-video and a runaway bill.
vast.ai exposes this as a plain CLI. You query the live offer book with a filter expression, rent a specific offer id, and tear it down when you’re done — all scriptable:
# Rank the live offer book: reliable single-GPU RTX cards with room on
# disk, sorted by dollars-per-hour ascending. dph = $/hour billed by the minute.
vastai search offers \
'reliability>0.98 gpu_name=RTX_3090 num_gpus=1 disk_space>40 rentable=true' \
-o 'dph' | head -5
# ID GPU dph disk reliability
# 8261234 RTX_3090 $0.194 120 0.991
# 8259871 RTX_3090 $0.201 200 0.988
# ...
vastai create instance 8261234 --image ghcr.io/example-org/ffmpeg-nvenc:latest --disk 50
vastai destroy instance 8261234 # ALWAYS — idle GPUs bill by the minute
The orchestrator wrapped that lifecycle around each pending row. The one rule that mattered was the finally: whatever happens, destroy the instance, because a leaked GPU quietly bills 24/7 at spot prices until someone notices the invoice.
def transcode_one(job: TranscodeJob) -> None:
# 1. cheapest interruptible RTX with NVENC and enough scratch disk
offer = cheapest_offer("reliability>0.98 gpu_name=RTX_3090 disk_space>40 dph<0.25")
instance = vast.create(offer.id, image="ghcr.io/example-org/ffmpeg-nvenc:latest")
try:
vast.exec(instance, pull_from_wasabi(job.source_key)) # original off Wasabi
vast.exec(instance, nvenc_ladder_cmd(job)) # 240p..1080p HLS on NVENC
vast.exec(instance, push_hls_to_wasabi(job.asset_id)) # renditions back to Wasabi
mark_done(job)
finally:
vast.destroy(instance) # non-negotiable: the meter never sleeps
The nvenc_ladder_cmd is where the real work happened, and it’s more than “encode small.” For adaptive-bitrate HLS, the player switches rungs mid-stream — jump from 480p to 720p the instant the network improves — and that only works if every rung’s keyframes land on the same timestamps, so a segment boundary in one rung is a segment boundary in all of them. That’s what the GOP flags enforce: -g 48 -keyint_min 48 -sc_threshold 0 pins a keyframe every 48 frames (2 seconds at 24fps) and disables scene-cut keyframes, which would otherwise fire at different moments in each rung and make the segments un-switchable. One ffmpeg invocation built the whole ladder, decoding once and scaling on the card with scale_cuda:
ffmpeg -hwaccel cuda -hwaccel_output_format cuda -i source.mp4 \
-filter_complex "[0:v]split=3[v1][v2][v3]; \
[v1]scale_cuda=w=426:h=240[v240]; \
[v2]scale_cuda=w=854:h=480[v480]; \
[v3]scale_cuda=w=1280:h=720[v720]" \
-map "[v240]" -c:v:0 h264_nvenc -preset p4 -tune hq -rc vbr -cq 27 -b:v:0 0 \
-g 48 -keyint_min 48 -sc_threshold 0 \
-map "[v480]" -c:v:1 h264_nvenc -preset p4 -tune hq -rc vbr -cq 25 -b:v:1 0 \
-g 48 -keyint_min 48 -sc_threshold 0 \
-map "[v720]" -c:v:2 h264_nvenc -preset p4 -tune hq -rc vbr -cq 23 -b:v:2 0 \
-g 48 -keyint_min 48 -sc_threshold 0 \
-map a:0 -map a:0 -map a:0 -c:a aac -b:a 128k -ac 2 \
-f hls -hls_time 6 -hls_playlist_type vod -hls_segment_type mpegts \
-master_pl_name master.m3u8 \
-var_stream_map "v:0,a:0 v:1,a:1 v:2,a:2" \
"stream_%v/index.m3u8"
Everything stayed resident on the card — decode, scale_cuda, and NVENC all on the GPU, so we never wasted the speedup shuttling frames across the memory bus to the CPU and back. NVENC did in a couple of minutes per rung what libx264 had been doing in twenty. A 1.2 GB, 20-minute HD source that was an afternoon’s work on a CPU worker came off a rented RTX in a handful of minutes.
Then I did the arithmetic that was the entire point of the experiment. An RTX 3090 at ~$0.19/hour is $0.0032/minute. The three-to-five-rung ladder for a 20-minute source finished in roughly 6 minutes of billed wall-clock including the pull and push — call it two cents a video. At 150 videos a day that’s about $3/day, under $100/month for the whole transcode fleet:
per-video : ~6 billed min x $0.0032/min ~= $0.019
per-day : 150 videos x $0.019 ~= $2.90
per-month : ~$90 (vs. a persistent cloud GPU idling at $250-400/mo,
vs. MediaConvert's per-output-minute pricing on 150/day)
That number was the prize. For the first time I had a measured marginal cost for a transcode — two cents — and it was small enough that the whole “150 videos a day will bury us” anxiety evaporated. On paper, the problem was solved.
Where it broke
Then I watched it run for a couple of weeks, and the paper met the weather.
The failure wasn’t the encode. NVENC was flawless. The failure was the tenancy. Four distinct gremlins, all rooted in the fact that these were interruptible machines — someone’s real gaming rig or a small host’s spare box — with no SLA behind them:
- Spot preemption mid-job. The defining failure. A host reboots, a higher bidder outbids my lease, or the owner sits down to game — and the instance is reclaimed with little or no warning, often three rungs into a five-rung ladder. There is no partial credit: the box is gone and so is every segment it had written to local disk but not yet pushed.
- Cold-provision latency. Renting a card wasn’t instant. Between
vastai createand a shell that could actually runffmpeg, I’d wait on the host to allocate, then pull the multi-hundred-megabyteffmpeg-nvencimage over a residential uplink. Tens of seconds on a good draw, a couple of minutes on a bad one — dead time I paid for on a two-cent job, and a tax I paid again on every retry. - Image and driver drift. The marketplace is a thousand different hosts running a thousand different setups. A card would come up with an NVIDIA driver too old for the CUDA build baked into my image, and
h264_nvencwould refuse to initialize — the encode failing not on the video but onCannot load libnvidia-encode.sobefore a single frame moved. - Advertised-vs-actual variance. Sometimes I’d rent a card listed as an RTX and get throughput that said otherwise — a thermal-throttling host, a contended PCIe bus, a downclocked card. The marketplace is only as honest as its long tail of hosts, and performance variance between two nominally-identical offers was wide enough that “6 minutes” was a median, not a promise.
Every one of those mapped to the same outcome in my jobs table: a pending row that had burned money and wall-clock and produced nothing, needing a full re-rent and a from-scratch redo. The orchestrator absorbed it with a bounded retry, but each retry re-paid the cold-provision tax and re-ran the entire ladder:
MAX_ATTEMPTS = 4
def dispatch(job: TranscodeJob) -> None:
for attempt in range(1, MAX_ATTEMPTS + 1):
try:
transcode_one(job) # rent -> pull -> ladder -> push -> destroy
return
except (InstancePreempted, ConnectionDropped, NvencInitFailed) as exc:
# no partial credit: a fresh box redoes every rung from zero
log.warning("job %s attempt %d/%d failed: %s",
job.id, attempt, MAX_ATTEMPTS, exc)
job.mark_pending() # release the row for another try
job.mark_stuck() # escalate: a human now babysits this one
sequenceDiagram participant O as Orchestrator participant G as Rented GPU participant DB as Jobs table O->>G: rent and start ladder G-->>O: 240p done, 480p done Note over G: owner reclaims the machine G--xO: connection dies mid encode O->>DB: job still pending, retry elsewhere Note over O: re-rent, re-pull, redo from scratch
Each interruption meant re-renting a fresh box, re-pulling the source from Wasabi, and redoing the whole ladder from zero — there was no partial credit. Individually survivable; my orchestrator retried. But the distribution was the problem. A pipeline that succeeds 80% of the time and silently stalls the other 20% doesn’t have an 80% problem — it has a “which videos are late today, and will anyone notice before a user does?” problem. “Your course video is ready” quietly became a coin flip, and for content people were paying to watch, a coin flip is not a service level. I found myself checking the jobs table by hand, which is the exact babysitting I’d fled MediaConvert’s arms to escape.
The outcome: proven, not dependable
I’ll be precise about what the experiment did and didn’t establish, because both halves mattered.
It proved the economics and the technology. GPU transcoding via NVENC was real, it was many times faster than our CPU path, and rented by the minute it cost cents per video — a marginal cost an NGO could carry at 150 videos a day without a second thought. That was a genuine, bankable finding.
It failed on dependability, for that era. Not a flaw in vast.ai’s idea — the marketplace model is elegant, and it’s matured a great deal since — but at that moment, for a service sitting in the critical path of paid content, the interruption rate and the performance variance were more than I could paper over with retries. I could not put my hand on my heart and tell the foundation that an uploaded video would be watchable by a known time. Cheap and fast, but not yet trustworthy — and trustworthy was the requirement, not a nice-to-have.
The whole experiment reduces to one decision gate — the encoder proved itself, the marketplace didn’t:
flowchart TD
J[Video job] --> R[Rent marketplace GPU]
R --> P{Instance survived}
P -->|yes| E[NVENC ladder in minutes for cents]
P -->|no| X[Preempted or drifted or throttled]
X --> RT{Retries left}
RT -->|yes| R
RT -->|no| S[Stuck row a human babysits]
E --> OK[Fast and cheap proven]
S --> BAD[Not dependable for paid content]The lesson: prototype the frontier, adopt on reliability
Here’s the judgment I carry into every “should we bet on the shiny new thing?” conversation now.
Prototype the frontier eagerly. Adopt it on reliability, not on excitement. Those are two different decisions, and the mistake is collapsing them into one. Prototyping vast.ai was unambiguously the right call — it was cheap, fast to stand up, and it bought me a number I could not have gotten any other way: the true marginal cost of a transcode. That number reframed every decision that followed in this series, including the one in the very next chapter. Prototyping the frontier is how you replace assumptions with measurements.
But adopting something into the critical path is a different bar, and the currency isn’t how impressive the demo was — it’s whether its worst day is one you can live with. vast.ai’s best day was spectacular. Its worst day was a paying user staring at a video that hadn’t finished encoding because a stranger in another timezone had sat down to play a game. For a system where “late” quietly becomes “broken,” I needed the boring virtue — predictability — that the exciting option couldn’t yet promise.
So I let the experiment do its real job. It didn’t hand me a production system; it handed me conviction. I now knew, with numbers, that a single consumer GPU running NVENC could chew through our entire daily upload flood for pocket change. The frontier had proven the physics. It just couldn’t yet promise the GPU would still be there when a video landed at 2 a.m.
Which left me with a proven appetite for GPU transcoding, a hard budget, and a reliability requirement the marketplace couldn’t meet. So I did something distinctly unconventional — the least fashionable decision in this entire series, and one of the ones I’m most sure about. That’s the next chapter.