Kuldeep Pisda
Payments Never Wait Behind a Transcode

Payments Never Wait Behind a Transcode

Splitting one Celery cluster into named RabbitMQ queues so instant payment webhooks never starve behind two-hour video transcodes on one shared backend.

14 min read

Payments Never Wait Behind a Transcode
Contents

The last chapter ended on a loose thread I promised to pick up: inside the one Django hub, plenty of work has no business running inside a web request. Sending a donor’s receipt, fanning out a notification, kicking off a video transcode — those all get handed off to Celery. And the moment you have both a two-hour transcode and a receipt a human is actively refreshing their inbox for, how you queue them stops being a detail and becomes the thing that decides whether the platform feels instant or broken.

This chapter is about that decision. It’s a small, unglamorous piece of configuration that I got wrong first, and getting it right is the difference between a payment that confirms in a blink and a payment that confirms twenty minutes later because it was stuck in line behind a movie.

First, what a task queue even is

When a user pays, CCAvenue calls a webhook on our backend — an HTTP request that says “this order is paid.” We could do all the follow-up work right there inside that request: mark the order paid, generate a PDF receipt, email it, unlock the course, update analytics. But the payment provider is waiting on the other end of that HTTP call, and it will retry or mark the call failed if we take too long. Doing slow work inside a request you’re supposed to answer in milliseconds is how you turn a fast system into a flaky one.

The fix is a task queue. Instead of doing the slow work now, the web request writes a little ticket — “generate receipt for order 8842” — drops it in a queue, and immediately answers the webhook with a 200. Something else, running separately, picks that ticket up a moment later and does the actual work. The user’s payment confirms instantly; the receipt lands a second later. The mental model is a restaurant: the waiter (web request) doesn’t cook your food, they clip the order to a rail and get back to taking orders; the kitchen (workers) cooks in the background.

Three pieces make that work, and it’s worth naming them because the rest of the chapter turns on how they interact:

  • Celery is the Python framework that defines those tickets (called tasks) and runs them. You decorate a function, call .delay(...) instead of calling it directly, and Celery handles the hand-off.
  • The broker is the thing that actually holds the tickets between “dropped off” and “picked up.” We use RabbitMQ — think of it as a post office that keeps an orderly line of messages and hands them out to whoever’s free. A queue here is literally a line: first in, first out.
  • Workers are separate long-running processes that sit next to RabbitMQ, pull tasks off the line, and execute them. You run as many as you have CPU and memory for.

For the first year, we ran the textbook setup: one queue, a pool of workers, everything goes in, workers pull it out. It’s the default for a reason — it’s dead simple and it works right up until your jobs stop resembling each other.

One more piece of vocabulary before the story, because two RabbitMQ words show up later and they matter. A queue can be durable — RabbitMQ writes it (and the messages in it) to disk, so a broker restart doesn’t vaporize a paid order’s receipt task. And there’s a dead-letter queue (DLQ): a sidings track where a message goes when it’s been rejected too many times or has expired, so a single poison task doesn’t loop forever or vanish silently — it lands somewhere you can inspect it. Celery on RabbitMQ gives you durable queues by default; the DLQ you wire up yourself, and I’ll show where.

The day the jobs stopped resembling each other

Early on, every background task was small and fast: send an email, sync a user from Keycloak, update a counter. A worker grabbed one, finished in under a second, grabbed the next. The line never really formed.

Then two things landed close together. Payments went live, which meant webhook follow-ups that had to feel instant. And in-house video came online — the platform’s whole reason to exist is long-form course video, and processing one course video (chopping it into streamable chunks at multiple qualities) takes minutes to hours on a CPU. Suddenly the same single queue held two species of work with nothing in common except that they were both “background.”

Here is the failure, and it’s a nasty one because nothing errors. A worker pulls a transcode task and disappears into it for ninety minutes. During those ninety minutes, a payment confirmed. Its receipt task is sitting in the queue — behind the transcode, or picked up by another equally-busy worker — and it just waits. The user sees “payment received” but no email, no unlocked course. Nothing crashed. Nothing logged an error. The system is simply doing exactly what a single line does when the person at the front is slow.

That’s head-of-line blocking: the item at the front of the line holds up everything behind it, regardless of how urgent the things behind it are. One slow checkout lane, and it doesn’t matter that you’re only buying gum. A single shared queue makes every task subject to the slowest task in front of it.

Two ways out, and why I rejected the first

Option one: throw more workers at it. If transcodes hog workers, add ten more workers so there’s always a free one for payments. This is the instinct, and it’s wrong for two reasons. First, it’s a probabilistic fix to a correctness problem — you’re just making it less likely a payment gets stuck, not impossible, and the day 150 videos upload at once (which happens on this platform) all your workers are transcoding again and payments starve anyway. Second, transcodes are CPU-bound and memory-hungry; you can’t just spin up ten more without a much bigger, more expensive machine. You’d be scaling the fleet to the heaviest job while trying to protect the lightest. Wrong lever.

The deeper problem with one shared queue isn’t throughput. It’s that unrelated workloads share a single fate. A flood of transcodes, a poison task that wedges a worker, a slow third-party email API — any one of them degrades all the others, because they’re all standing in the same line.

Option two: give each kind of work its own line. Instead of one queue, define named queues by workload — payments, email, index, transcode — and route each task to its own. Then run dedicated worker pools, each draining only certain queues. The payments workers never even look at the transcode queue, so they are structurally incapable of getting stuck behind a video. This is the one I took.

One broker, one exchange, four named lines — and worker pools that each subscribe to only the lines they’re allowed to touch. This is the entire shape of the fix:

The routing is the whole trick, and in Celery it’s a table. Tasks are named by their dotted Python path, so you route by glob; anything that doesn’t match a rule falls through to the default queue, which is why the map is exhaustive on purpose — an unrouted task silently landing on payments would defeat the whole point.

# config/celery.py
from celery import Celery
from kombu import Exchange, Queue

app = Celery("happy_thoughts")
app.config_from_object("django.conf:settings", namespace="CELERY")

# Every queue is durable (survives a broker restart) and the paid-order
# path must never lose a message, so we declare them explicitly rather
# than let Celery auto-create them with defaults we didn't choose.
default_exchange = Exchange("ht", type="direct", durable=True)

app.conf.task_queues = (
    Queue("payments",  default_exchange, routing_key="payments",  durable=True),
    Queue("email",     default_exchange, routing_key="email",     durable=True),
    Queue("transcode", default_exchange, routing_key="transcode", durable=True,
          # a transcode that keeps failing gets dead-lettered, not retried forever
          queue_arguments={
              "x-dead-letter-exchange": "ht.dlx",
              "x-dead-letter-routing-key": "transcode.dead",
          }),
    Queue("default",   default_exchange, routing_key="default",   durable=True),
)
app.conf.task_default_queue = "default"
app.conf.task_default_exchange = "ht"
app.conf.task_default_routing_key = "default"

# One rule per workload. Order doesn't matter; specificity does — these
# are non-overlapping globs, and the unmatched remainder falls to `default`.
app.conf.task_routes = {
    "payments.tasks.*":  {"queue": "payments"},
    "notify.tasks.*":    {"queue": "email"},
    "media.tasks.*":     {"queue": "transcode"},
    "search.tasks.*":    {"queue": "default"},
}

# --- the two settings that actually make isolation behave ---

# acks_late: a task is acknowledged AFTER it finishes, not when the worker
# picks it up. If a transcode worker is OOM-killed mid-job, RabbitMQ never
# saw an ack, so it re-delivers the task to another worker instead of
# dropping it. The cost: tasks must be idempotent (a redelivered receipt
# must not double-charge or double-email). Worth it for anything you can't
# afford to silently lose.
app.conf.task_acks_late = True

# prefetch: how many messages a worker reserves up-front before it has even
# started them. The default (4) is a throughput win for sub-second tasks and
# a catastrophe for hour-long ones — see the transcode note below. We set the
# global default low and let the heavy pool pin it to 1 at the CLI.
app.conf.worker_prefetch_multiplier = 1

# a worker that dies takes at most one in-flight job down with it, and that
# job is redelivered — because acks are late and prefetch is 1.
app.conf.task_reject_on_worker_lost = True

Then you start separate worker processes, each told which queues it’s allowed to drain with -Q, and — critically — with a concurrency and pool type chosen for that queue’s latency class. This is where the isolation stops being a config file and becomes four OS processes that physically cannot interfere:

# FAST LANE — payments + receipts. Latency-critical, sub-second, I/O-bound
# (DB write, PDF render, SMTP handoff). Many green-thread-ish slots is fine;
# they spend most of their time waiting on the network, so 8 prefork slots
# stay cheap. prefetch 1 so a momentary SMTP stall can't let one worker
# hoard a paid order's receipt behind it.
celery -A config worker -Q payments,email -c 8 \
    --prefetch-multiplier 1 -n fast@%h

# DEFAULT LANE — search reindex, housekeeping, Keycloak user sync. Not urgent,
# not heavy. Middle concurrency, standard prefork.
celery -A config worker -Q default -c 4 -n default@%h

# HEAVY LANE — transcodes only. CPU/GPU-bound, minutes to hours each. Few slots
# (2) because each one pins real hardware; --pool=solo per box when the encode
# owns the whole GPU. prefetch 1 is non-negotiable here (next paragraph).
celery -A config worker -Q transcode -c 2 --pool=prefork \
    --prefetch-multiplier 1 -n media@%h

The concurrency numbers aren’t guesses — they’re one per latency class. The fast pool gets eight slots because payment/email work is I/O-bound and idle-heavy; eight of them barely touch a CPU and they turn over in milliseconds. The heavy pool gets two because each transcode saturates a core (or, later in this series, an entire GPU), so a third concurrent job wouldn’t run faster — it would just thrash. A transcode storm can peg the heavy lane for hours and the fast lane never notices, because those eight workers are not subscribed to transcode at the broker level. There is no shared resource left to contend on.

That --prefetch-multiplier 1 on the heavy pool is the one non-obvious gotcha, and it cost me an afternoon. By default a Celery worker greedily prefetches a batch of tasks off the queue and reserves them, betting they’ll each be quick. For sub-second tasks that batching is a real speedup. For hour-long transcodes it’s a disaster: one worker grabs four transcodes at once, works the first, and the other three sit reserved — invisible to the idle worker next to it that could have taken one. So you get one pegged worker with a private backlog while its neighbour spins idle. Prefetch 1 tells long-task workers to reserve exactly one job and nothing more, so heavy work spreads evenly instead of clumping on one unlucky process. Combined with acks_late, it also means a crashed transcode worker forfeits exactly one job — which RabbitMQ then hands to a healthy worker — instead of taking a reserved batch of four down with it.

Routing to a queue is only half of it; the task itself declares its own retry and failure policy, so a transient failure recovers on its own and a permanent one gets dead-lettered instead of hammering a broken dependency forever. Here’s the media transcode task with the whole contract on it:

# media/tasks.py
from celery import shared_task
from botocore.exceptions import EndpointConnectionError

@shared_task(
    queue="transcode",           # belt-and-suspenders alongside task_routes
    bind=True,
    acks_late=True,              # don't ack until the encode actually finishes
    # a flaky S3/Wasabi endpoint or a dropped GPU-box connection is transient —
    # retry it. A malformed source file is NOT — let that one fail to the DLQ.
    autoretry_for=(EndpointConnectionError, ConnectionError),
    retry_backoff=10,            # 10s, 20s, 40s, ... exponential
    retry_backoff_max=600,       # cap the wait at 10 minutes
    retry_jitter=True,           # spread retries so a broker blip doesn't sync them
    max_retries=6,
)
def transcode_to_hls(self, upload_id: int) -> None:
    """Pull a source upload, encode HLS renditions, push to object storage.

    Idempotent by design: re-running for the same upload_id overwrites the
    same HLS prefix, so an acks_late redelivery is safe.
    """
    upload = VideoUpload.objects.select_related("course").get(pk=upload_id)
    render_hls_ladder(upload)     # ffmpeg/NVENC; the media chapter goes deep here

The payments task carries the same shape but a tighter retry budget — a payment webhook that we can’t process is a customer staring at a spinner, so it backs off fast and hands off to a human-visible reconciliation queue rather than retrying for ten minutes. The point is that every task states, in its own decorator, what “transient” means for it and where it goes when it’s out of retries. A message that exhausts max_retries on the transcode queue doesn’t disappear and doesn’t loop — the x-dead-letter-exchange we declared routes it to a transcode.dead DLQ, where a nightly job (and a Grafana alert on DLQ depth) can surface “these three uploads never encoded” instead of them evaporating into a log line nobody reads.

What it bought us

The payment path is the headline. Before, a receipt could land anywhere from instant to twenty-plus minutes late depending entirely on what the shared workers happened to be chewing on. After, receipts and course-unlocks land in about a second, flat, regardless of how many videos are transcoding — because the workers doing them never see a transcode. During the daily upload wave of 150-plus videos, the heavy lane pegs for hours and the payments lane stays completely idle-fast. The two workloads no longer share a fate.

The second, quieter win is operational legibility, which matters more than it sounds on a tiny NGO team. Each queue has its own depth you can watch. When the transcode queue backs up, that’s a capacity signal — buy more transcode muscle — and it’s cleanly separated from a backup in email, which is a third-party signal — the mail provider is slow. On one shared queue those two very different problems look identical: “the queue is deep.” Named queues turn one undifferentiated number into a per-workload dashboard, and they let you scale exactly the lane that’s hurting instead of the whole fleet.

There’s a nice failure-isolation bonus too. A poison task that wedges a worker can only wedge workers on its queue. A bug in the search reindexer can take down the index lane and payments won’t even flinch.

The lesson

Isolate workloads by latency-criticality, not by what’s technically convenient. The right question for any two background jobs isn’t “are they both async?” — it’s “if one floods, is it acceptable for the other to wait?” If the answer is no, they do not belong in the same queue. A payment webhook and a video transcode are both “background work” in the same way an ambulance and a cement truck are both “traffic”: lumping them into one lane optimizes for the wrong thing.

A single shared queue is a single shared fate. Everything in it degrades together, and the slowest, heaviest, flakiest task sets the floor for all the rest. Splitting into named queues with dedicated workers costs you a routing table and a few extra worker commands — genuinely a day of work — and in return your urgent path becomes structurally immune to your slow path. That’s a rare trade: a small amount of config that buys a guarantee instead of a probability.

Those transcode queues I’ve been using as the villain here are about to become the main character — the media pipeline leans on this isolation harder than anything else on the platform. But before we get to moving terabytes of video, the same one backend has to do two genuinely hard things under load: hold a physical booking system correct when two people grab the last slot at once, and let a non-profit define its own org chart and permissions from a UI. Two hard apps, one backend — that’s next.