Contents
The FastAPI proxy had done exactly what I built it to do. It kept the course screen fast, bolted CCAvenue payments onto the edge, and carried us from 2,000 users to roughly 6,000 in three months with about one in ten paying. But the proxy fixed the wrong end of the pipe. It made the metadata fast. It did nothing for the thing users actually came for: the video. And the video had a problem I could not cache my way out of.
The complaint arrived almost every week, in Hindi and in English, always a version of the same sentence: I watched one video and my whole day’s data was gone. They were not exaggerating. A single twenty-minute course video on the Happy Thoughts platform was 1.2 GB in HD, and a typical prepaid data pack in India at the time was around 1 to 1.5 GB for a day. One lesson could cost a user their entire day of internet.
“Almost every week” is the kind of phrase that feels true and quietly lies. Before I let it drive a decision, I made it a number. Our support inbox fed a plain Django SupportTicket model, so the cheapest possible instrumentation was a tag. Ops started stamping every one of these with a data-burn slug the week the pattern was obvious, and one query told me whether it was a trickle or a trend:
-- One tag, one query. The whole "measurement" was an afternoon of ops discipline.
SELECT date_trunc('week', t.created_at)::date AS week,
count(*) AS data_burn_tickets
FROM support_ticket t
JOIN support_ticket_tags st ON st.ticket_id = t.id
JOIN tag ON tag.id = st.tag_id
WHERE tag.slug = 'data-burn'
AND t.created_at >= now() - interval '90 days'
GROUP BY 1
ORDER BY 1;
week | data_burn_tickets
------------+-------------------
2022-05-30 | 9
2022-06-06 | 14
2022-06-13 | 11
2022-06-20 | 17
2022-06-27 | 13
(5 rows)
Nine to seventeen tickets a week, steady, on a base of ~6,000 users where only a fraction ever open a support thread at all. That is not noise; a written complaint is the loud tip of a silent iceberg of people who just close the app. The tag turned a feeling into a slope I could watch — and, later, watch fall after the bandaid shipped.
The tickets told me people were angry. They did not tell me how much data a lesson actually cost, because a user’s “my whole pack” is itself an estimate. For that I had the player emit a tiny beacon on ended (and on unload) with the bytes it had pulled, into a playback_event table, and let Postgres do the arithmetic per lesson:
SELECT lesson_id,
count(*) AS plays,
round(avg(bytes_transferred) / 1e6) AS avg_mb,
round(percentile_cont(0.9)
WITHIN GROUP (ORDER BY bytes_transferred) / 1e6) AS p90_mb
FROM playback_event
WHERE event = 'ended'
AND created_at >= now() - interval '30 days'
GROUP BY lesson_id
ORDER BY avg_mb DESC
LIMIT 3;
lesson_id | plays | avg_mb | p90_mb
-----------+-------+--------+--------
412 | 803 | 1180 | 1210
389 | 611 | 1174 | 1208
377 | 994 | 1096 | 1205
There was the anecdote, confirmed in bytes: a single completed play of lesson 412 moved ~1.18 GB across a metered connection, p90 essentially the full file. The complaint was not a perception problem. The pipe really was that fat.
Every engineering instinct I had said fix this now. I didn’t. For a meaningful stretch of the product’s life we shipped that painful, broken-feeling video on purpose. This chapter is the case for that decision, because knowing what not to build is the harder and more valuable half of the job, and premature infrastructure is one of the quieter ways a young product dies.
Why a raw HD file is so enormous
Start with where 1.2 GB even comes from, because the number is not arbitrary.
A video is just a flip-book: a sequence of still frames shown fast enough to look like motion, usually 25 to 30 frames every second. A single 1080p frame is 1920 by 1080 pixels, a little over two million of them, and each pixel needs colour information. Store every frame in full and a few minutes of footage is tens of gigabytes. Nobody does that. A codec — the compressor, H.264 in our case — shrinks it with one central trick: most of the frame barely changes from one moment to the next, so instead of storing every frame whole, it stores one full frame and then only the differences until the picture changes enough to need a fresh full frame.
The dial that controls the result is bitrate: how many bits the codec is allowed to spend per second of video. Higher bitrate means more detail preserved and a bigger file; lower bitrate means a smaller file and, past a point, visible mush. The arithmetic is unforgiving. Our content team exported lessons straight out of their editing tool as 1080p H.264 and uploaded the raw file. Twenty minutes at 1.2 GB works out to about 8 megabits per second — a perfectly ordinary bitrate for an untouched HD export, and a catastrophic one to hand to a phone on a metered plan.
8 Mbps x 60 sec x 20 min = 9,600 megabits
9,600 megabits / 8 = 1,200 megabytes ~ 1.2 GB
That is the whole mystery. There was no bug. The file was exactly as large as an uncompressed-intent HD export should be. The bug was that we shipped that file, untouched, to the worst possible network.
What a mobile data pack means in India
To a founder in a metro on unlimited fibre, a gigabyte is background noise. To our actual audience it was a budget. Most Happy Thoughts users were on prepaid mobile plans with a daily data cap — you buy, say, 1.5 GB for the day, and once you cross it your speed collapses to a throttled crawl that makes video unwatchable until midnight resets the counter. Data was not abstract to these users. It was a resource they rationed, in a household that often shared one connection, frequently on a 3G or weak-4G signal in a smaller town.
Put the two numbers next to each other and the product is broken by arithmetic, not by any bug:
Cost of one lesson: 1.2 GB (measured p90 above)
Daily "unlimited" pack: 1.5 GB/day, then throttled to a crawl
-> lessons before throttle: 1.5 / 1.2 = 1.25 lessons
One lesson, and the day is effectively over.
Cheaper prepaid tier: ~2 GB total for the month (validity ~28 days)
-> lessons the whole pack buys: 2 / 1.2 = 1.6 lessons
A learner on the budget tier cannot finish a second lesson all month
without buying more data than the course itself is worth.
So the 8 Mbps export did not just cost money. It vaporised the day. A learner would open one lesson before work, and by the time the video ended, the pack that was supposed to last until night was spent — no messaging, no other apps, nothing. For a devotional, mobile-first audience we existed to serve, that was not a rough edge. It was a betrayal, roughly once per lesson.
What a naive whole-file setup actually does
Here is the part worth understanding, because it is exactly what we had. There was no transcoding anywhere in the path — no encode ladder, no smaller renditions, no adaptive switching, not even a download-on-Wi-Fi toggle. Just the raw 1080p export sitting behind a CDN.
When a player hits a plain file like that, it does a progressive download: it pulls the single file from the beginning, byte after byte, and starts playing once it has enough buffered. There is exactly one version of the video, so a 12,000-rupee phone on flaky 3G is handed the identical 8 Mbps stream as a flagship on Wi-Fi — the network has no smaller option to fall back to. And because it is one linear file, a user who watches three minutes and leaves has often already pulled far more than three minutes of data down the pipe. No chunking, no quality tiers, no negotiation. One fat file, one bitrate, everyone.
flowchart LR A[Content team exports 1080p H.264] --> B[Raw 1.2 GB file in object storage] B --> C[CDN] C -->|whole file at 8 Mbps| D[Phone on mobile data] D --> E[1.2 GB pulled down] E --> F[Daily 1 to 1.5 GB pack gone]
The fix is not a mystery either, and I could draw it in my sleep. You build an adaptive-bitrate pipeline: transcode each upload into a ladder of renditions — 240p, 360p, 480p, 720p, 1080p — chop each into small chunks, and let the player pick the rendition that fits the network moment to moment, so a phone on 3G quietly pulls the 360p chunks and nobody’s pack dies. That is real, correct, and exactly what we would eventually build. The judgment was not in knowing the answer. It was in knowing it was the wrong week to build it.
The obvious fix was a trap
An adaptive pipeline is not a feature you add. It is a fleet of ffmpeg workers pulling off a queue, storage for every rendition of every video, a packaging step, a player that switches streams as the network changes, and a CDN in front of all of it. It is weeks of build followed by a permanent operations surface and a monthly bill that never stops arriving.
I did the back-of-the-envelope honestly, as if I were about to start Monday, and it was sobering for a team where “engineering” was one person most weeks:
BUILD ESTIMATE — adaptive HLS, done properly (one engineer, 2022)
-----------------------------------------------------------------
ffmpeg ABR ladder + keyframe-aligned segments, tuned ~1.5 wk
(-g/-keyint_min/-sc_threshold pinned so renditions are switchable)
Celery queue + worker autoscaling, retries, acks_late ~1.0 wk
S3/CDN layout, signed URLs, cache + invalidation ~1.0 wk
Player: HLS.js, ABR switching, resume, error states ~1.0 wk
Upload -> transcode -> publish state machine + admin ~1.0 wk
Reprocessing the existing back catalogue ~0.5 wk
Buffer for the unknowns transcoding always hides ~1.0 wk
-----------------------------------------------------------------
TOTAL ~7 engineer-weeks
+ a permanent ops surface, forever
Seven weeks was the optimistic read, and it undersells the real cost, which is not the seven weeks — it is every week after them. (The catalogue-reprocessing line alone later exploded into the entire transcoding saga this series spends three chapters on; the estimate had no idea what it was signing up for.)
And at the moment the complaint was loudest, what did we actually have? Roughly 6,000 users, about one in ten paying, the whole thing held together by a FastAPI proxy hacked in front of a headless CMS. Building a real streaming stack on top of that was not a feature decision. It was a bet — and the thing it secretly bet on was that people would pay for these courses, repeatably, in numbers that justified the infrastructure.
That was precisely the question we had not yet earned an answer to. Ten percent paying was encouraging, but it was young and thin: three months of data on a platform that had been a content experiment a quarter earlier. If I built the pipeline and the answer turned out to be no, this stays a nice-to-have nobody pays for, I would own a beautiful adaptive-streaming stack bolted to a product that did not need to exist.
So I wrote the deferral down as an explicit rule, not a vibe, with a number attached so it could not quietly drift:
DECISION RULE — build the streaming pipeline WHEN, and only when:
(a) paying learners are undeniable, not encouraging:
>= 25% of active users paying, held steady for >= 3 months
(not the current thin, 3-month-old ~10%), AND
(b) the pain is measured, not just loud:
the data-burn ticket slope is flat-or-rising after the bandaid.
UNTIL BOTH HOLD:
buy the pain down by hand (warn + hand re-exports),
spend the freed weeks on the conversion loop that tests (a).
COST OF BEING WRONG:
defer wrongly -> a few angry weeks, fully reversible.
build wrongly -> ~7 wks sunk + permanent ops on a product still auditioning.
The asymmetry in that last block is the whole argument. A wrong deferral costs a reversible few weeks of visible pain; a wrong build costs seven irreversible weeks plus a bill that arrives every month whether the product lives or not.
That is the trap of premature infrastructure, and it does not kill you in a fire. It drains you slowly: the maintenance, the on-call attention, the monthly egress, and above all the opportunity cost of the feature you didn’t build because you were busy tuning an encode ladder for a product still auditioning for the right to live. The scarce resource on that team was never compute. It was my attention — and spent on transcoding, it could not be spent on the one thing that would tell me whether transcoding was ever worth building.
flowchart TD
A[Loud painful gap spotted] --> B{Is the metric that funds the fix proven}
B -->|No, still validating| C[Buy the pain down cheaply<br/>warn on metered data]
B -->|Yes, demand undeniable| D[Build the real pipeline]
C --> E[Spend scarce attention on the validation loop]
E --> BWhat we built instead
So the attention went to the validation loop, not the video. Payments through CCAvenue so people could actually pay. The caching proxy that kept the CMS fast enough that they did not abandon the funnel before they got there. The path from free listener to paying learner, instrumented so I could watch it convert. Every week I did not spend on transcoding was a week spent finding out whether transcoding would ever pay for itself. That is the trade nobody claps for: there was no demo, and the embarrassing gap stayed visible while I poured the work into a question mark.
But deferring a problem is not the same as ignoring it, and the difference is the whole discipline. We kept the complaint in plain sight — tracked it, read it, let it stay uncomfortable — and we bought the pain down with the cheapest possible bandaid, which was not a pipeline. It was an afternoon: detect a data-conscious connection in the browser and warn before a 1.2 GB stream starts.
// Not a transcoding pipeline. An afternoon.
function shouldWarnBeforePlay() {
const conn = navigator.connection;
if (!conn) return false; // old browser: don't nag
return conn.saveData || /(^|slow-)2g$|3g/.test(conn.effectiveType);
}
if (shouldWarnBeforePlay()) {
showBanner(
"This video is about 1.2 GB in HD. On mobile data it can use a " +
"full day's pack - watch on Wi-Fi if you can."
);
}
This did not make the file one byte smaller. It made the surprise smaller. It converted a betrayal — my data just vanished — into a choice — I was warned and I pressed play anyway. For the handful of worst offenders, the content team also re-exported a smaller file by hand. Ugly, manual, not a system, and completely fine: buying down pain by hand is allowed while you are still validating; automating it prematurely is not. Empathy is cheap and infrastructure is expensive, so spend the cheap thing first. The angry-complaint rate dropped enough to buy the runway we actually needed, and the runway was the point.
When the gap finally earned its fix
The rule I settled on, and still use: you build the infrastructure when the metric that justifies it is undeniable, not when the pain is loud. The pain was loud from day one. What I was waiting on was the paying-user number to stop being arguable. Once demand was genuinely proven, the video problem had not gone anywhere — but now it was a problem worth solving properly, funded by revenue I could point at.
And restraint bought more than the deferred build cost. Had I built the pipeline in month three, I would have built it on assumptions instead of traffic, and I would have built the wrong version. The AWS-everything architecture that looks obviously correct on a whiteboard hid a failure mode only real load revealed: one ordinary day, about 5,000 users watching, roughly 1,500 USD of CloudFront egress in 24 hours — for a non-profit. Building early would have meant committing to that cost curve before I understood it. Waiting meant I met that lesson with revenue in the bank and a clear head, not with a half-validated product and a surprise invoice.
The takeaway for the next engineer
Knowing what not to build is harder than building, because the not-building is invisible while the pain is loud and specific. Anyone can point at the 1.2 GB video and say fix it. The judgment is in the next question: what does this feature’s return actually depend on, and have we answered it yet? If you haven’t, resist the pipeline. Buy the pain down for an afternoon’s work, keep the gap honestly in view, and spend your scarce attention on the question the whole bet rides on.
Premature infrastructure rarely kills a startup in one dramatic blow. It kills it slowly, one well-built, beautifully engineered, entirely unjustified system at a time. The restraint to leave a real problem visibly unsolved, on purpose, while you go answer the question that tells you whether it is worth solving, is not neglect. On a young product, it is the most senior thing you can do — and soon enough, the numbers came back and said now, build it right.