Splitting Celery Workers by Queue, Not by Scale

Sun, 15 Mar 2026 10:00:00 +0530

Here is the bug that never happened, which is the whole point.

A user opens the app, taps “log in”, and waits for a six-digit code to land on their phone. At the same moment, a clinic receptionist somewhere else hits “export” on a year of billing history — a fat Excel file that takes a while to build. On a naive Celery setup, those two tasks go into the same queue and get picked up by the same pool of workers. If the export grabbed a worker first, the OTP sits behind it. The user stares at a blank code field. To them, login is broken. Nothing threw an exception. The export was doing exactly what it was told.

Payments Never Wait Behind a Transcode

Sat, 28 Feb 2026 10:00:00 +0530

The last chapter ended on a loose thread I promised to pick up: inside the one Django hub, plenty of work has no business running inside a web request. Sending a donor’s receipt, fanning out a notification, kicking off a video transcode — those all get handed off to Celery. And the moment you have both a two-hour transcode and a receipt a human is actively refreshing their inbox for, how you queue them stops being a detail and becomes the thing that decides whether the platform feels instant or broken.

Task-Queues on Kuldeep Pisda

Splitting Celery Workers by Queue, Not by Scale

Payments Never Wait Behind a Transcode