Splitting Celery Workers by Queue, Not by Scale

Here is the bug that never happened, which is the whole point.

A user opens the app, taps “log in”, and waits for a six-digit code to land on their phone. At the same moment, a clinic receptionist somewhere else hits “export” on a year of billing history — a fat Excel file that takes a while to build. On a naive Celery setup, those two tasks go into the same queue and get picked up by the same pool of workers. If the export grabbed a worker first, the OTP sits behind it. The user stares at a blank code field. To them, login is broken. Nothing threw an exception. The export was doing exactly what it was told.

I built and ran a veterinary clinic platform solo for years — a veterinary clinic platform — and this was one of the earliest infra decisions I made that I still think was correct. I split Celery workers by queue, not by scale. And I did it on a deliberately tiny cluster, which is the part that makes it interesting.

Throughput is the wrong lens#

When people reach for multiple Celery queues, the reason is almost always throughput. “We have too many tasks, one worker pool can’t keep up, let’s shard the load.” That’s a real problem and queues do solve it. But it’s not the problem I had.

My problem was latency isolation. The tasks in my system were wildly unequal in urgency, and they were unequal in a way that had nothing to do with how many of them there were. Look at the actual workloads:

• OTP SMS — a human is actively waiting on it. Every second counts. There aren’t many of them.
• Bulk Excel import/export — nobody is staring at it. It’s slow by nature. It can take thirty seconds and that’s fine.
• Wallet / cashback jobs — money math that must be correct, but not millisecond-urgent.
• Scheduled reminders — big fan-out batches that fire on a cron, not on a user’s click.

The trap in a single shared pool is head-of-line blocking. A Celery worker doesn’t just grab one task and stop — with the default prefetch it pulls several off the broker and holds them. So a long export doesn’t only block the worker it’s running on; it can be sitting in front of an OTP that got prefetched right behind it. The urgent task pays for the slow task’s sins. Throughput was never the issue. The issue was that I was mixing “a human is waiting” work with “this can take a minute” work in the same line.

The cluster was tiny on purpose#

Here’s what makes this a genuine decision and not just cargo-culted best practice: I was not running a big fleet. The whole thing lived on a small footprint — a single small instance for the API, a handful of gevent workers, one Redis doing double duty as broker and result backend. This was never a “we have twelve nodes, let’s distribute” situation.

So the conventional read of “split your queues” — that you do it once you’ve outgrown one pool — didn’t apply. By throughput logic I had no reason to split at all. One worker could comfortably chew through the volume. I split anyway, because the decision wasn’t about volume. It was about making sure the one class of task a user is physically waiting on never shares a line with the one class of task that’s slow by design.

That reframing is the whole post: queue design is a latency decision as much as a throughput one. And latency isolation is worth doing even — especially — when you’re small, because a small team can’t afford a mysterious “why is login slow sometimes” ticket that only shows up under load.

Five queues, routed in one place#

I ended up with five named queues, each mapped to the kind of work that belongs on it: send_text_messages, reminders_cron, export, import, and wallet. The routing lives in exactly one place so there’s no guessing where a task lands.

# petapp_api/celery.py
app.conf.task_routes = {
    # a human is waiting on this one
    "auth_profiles.tasks.send_otp":        {"queue": "send_text_messages"},

    # slow by nature, nobody is staring at it
    "export.tasks.export_bills_xlsx":      {"queue": "export"},
    "clinic.tasks.import_medicines":       {"queue": "import"},

    # correctness matters, urgency doesn't
    "wallet.tasks.apply_cashback":         {"queue": "wallet"},

    # scheduled fan-out, fires on cron not on a click
    "clinic.tasks.send_vaccine_reminders": {"queue": "reminders_cron"},
}

Centralizing routing in a single task_routes dict matters more than it looks. The alternative — sprinkling queue= into every apply_async call — means the day you want to move a task to a different lane, you’re grepping the codebase. With one dict, the entire urgency map of the system is one screen you can read top to bottom.

One container per queue#

The routing is only half of it. A queue with no dedicated worker listening to it is just a list that fills up. The other half is running one worker container per queue, each pinned to its lane with -Q:

# docker-compose-celery.yml (trimmed)
services:
  worker-sms:
    image: kdpisda/vetclinic-celery
    command: celery -A petapp_api worker -Q send_text_messages -c 2

  worker-export:
    image: kdpisda/vetclinic-celery
    command: celery -A petapp_api worker -Q export -c 1

  worker-import:
    image: kdpisda/vetclinic-celery
    command: celery -A petapp_api worker -Q import -c 1

  worker-wallet:
    image: kdpisda/vetclinic-celery
    command: celery -A petapp_api worker -Q wallet -c 1

  worker-cron:
    image: kdpisda/vetclinic-celery
    command: celery -A petapp_api worker -Q reminders_cron -c 1

Same image every time — the API, all the workers, and beat run off one Docker artifact, which kept the ops surface small. The only thing that varies per container is the -Q flag naming which queue it drains.

Now the isolation is physical, not cooperative. The export worker can be pegged at 100%, prefetch-blocked, chewing on a giant spreadsheet — and it literally cannot touch the SMS worker’s process. send_text_messages has its own prefetch, its own event loop, its own CPU time. A stuck export starves the export queue and only the export queue. The OTP path stays clear because it isn’t sharing anything with the work that’s allowed to be slow.

The reminders were the sneaky one#

The subtle case was reminders, because those are also SMS. Beat drives an engagement loop — daily vaccine, deworming and prescription reminders in the morning, a pet-birthday job at midnight, a weekend “post a picture of your pet” nudge, a weekly credits reminder. All of those send text messages too.

The instinct would be to route them onto send_text_messages since that’s “the SMS queue.” That would have been the mistake. A morning reminder run enqueues a big batch all at once — exactly the kind of fan-out that could sit in front of a login OTP. So the reminder jobs go through reminders_cron, a completely separate worker, and only OTPs live on send_text_messages. Same medium, different urgency, different lane. The queue name isn’t “SMS” — it’s “a human is waiting on this specific message.”

Where I’d stop#

I want to be honest about the cost, because “split everything” is its own failure mode. Five worker containers on a small box each carry a little memory overhead, and each one is a process to supervise. If I’d made a queue per task instead of per urgency class, I’d have paid that overhead for no benefit and buried the routing map under noise.

The rule I settled on: split by how much a delay hurts, not by what the task does. Group tasks into a handful of urgency tiers, give each tier a queue, and stop there. Five was right for this system. Nine would have been vanity. One would have let a bulk export delay a login code.

What it actually bought#

Almost nothing to set up — a routing dict and a few extra command: lines — and it retired an entire category of bug before it could exist. In years of running this thing solo I never once had to chase a “login is slow sometimes” report, because the one thing that made login feel slow was structurally impossible.

That’s the part I’d tell my earlier self. The move looked like an infrastructure decision — worker containers, queue routing, broker config. It was actually a product decision wearing infra clothes: I decided which of my users’ waits were allowed to be interrupted, and I encoded that judgment into the topology so it couldn’t be undone by accident. Splitting Celery by queue instead of by scale is one of the cheapest ways I know to make that call once and have it hold.