Contents
In the last chapter I poured the walls between our Celery workloads so a slow transcode could never delay a donor’s payment — isolation by named queue, one worker per latency class. That was about fairness under load: making sure nobody waits behind the wrong thing. This chapter is about the harder cousin of that problem — correctness under load. Not “who goes first,” but “what happens when two people try to take the exact same thing at the exact same instant, and both of them are right to think it’s available.”
The new problem: Ambarsthan, and the last seat
The same one Django backend that runs the Happy Thoughts learning platform also runs Ambarsthan — a Tejgyan facility where people book stays, sit for sessions, and reserve seats. Bookings look boring until you look closely. A meditation session has, say, forty seats. A room is free for a given set of dates or it isn’t. On a quiet day this is trivial. The trouble is that our load is not evenly spread — it spikes, hard, the moment registrations open for a popular session. Hundreds of people hit Book now inside the same few seconds, and some of them are competing for the last seat or the same room on the same night.
That is the whole game: the moment demand and supply collide on a single unit, the code that decides who gets it runs many times concurrently — overlapping in time, on different worker processes, against the same rows. And the naive way every one of us writes this the first time is quietly, catastrophically wrong.
Crash course: what a race condition actually is
A race condition is when the correctness of your program depends on the timing of things you don’t control — the order in which two overlapping operations happen to interleave. Change the timing by a millisecond and you get a different, sometimes broken, result.
Here is the naive booking flow, the one that feels obviously correct:
# DON'T do this — it races.
def book_seat(session_id, user):
taken = Booking.objects.filter(session_id=session_id).count()
if taken < SEAT_LIMIT: # check
Booking.objects.create( # then act
session_id=session_id, user=user,
)
Read the count, and if there’s room, insert. Check-then-act. It works perfectly every time you test it alone, which is exactly why it survives to production.
Now run it twice at the same instant with one seat left. Both requests read the count — both see thirty-nine of forty taken. Both conclude there’s room. Both insert. You have just sold forty-one seats in a forty-seat room, and no error fired, no exception was logged, nothing crashed. The code did precisely what it said. The gap between the count() and the create() is a window, and under load, two requests climb through it side by side.
Here is that failure as a sequence — the two readers who both see “one left”:
sequenceDiagram
participant A as User A
participant B as User B
participant DB as Database
A->>DB: how many seats taken
DB-->>A: thirty nine of forty
B->>DB: how many seats taken
DB-->>B: thirty nine of forty
A->>DB: insert booking
B->>DB: insert booking
Note over DB: both commit, seat sold twiceYou cannot fix this by reading more carefully or adding a second check. Any check you add lives in the same window. The problem isn’t the check — it’s that nothing is stopping the two requests from being inside the decision at the same time.
The two tools the database gives you
The fix is to stop treating “is this slot free?” as a question the application asks and then acts on. It has to be a single, indivisible operation that the database itself serializes. Two mechanisms do this, and they solve two different shapes of the problem — and Ambarsthan needs both, because it sells two different shapes of scarcity. A meditation session is counted inventory: forty interchangeable seats. A room is geometric inventory: one physical room, and the only question is whether two stays overlap in time. Here’s the whole models file that carries both:
# ambarsthan/bookings/models.py
import uuid
from django.contrib.postgres.constraints import ExclusionConstraint
from django.contrib.postgres.fields import DateTimeRangeField, RangeOperators
from django.db import models
from django.db.models import F, Func, Q, Value
class BookingStatus(models.TextChoices):
PENDING = "pending", "Pending"
CONFIRMED = "confirmed", "Confirmed"
CANCELLED = "cancelled", "Cancelled"
class Room(models.Model):
id = models.UUIDField(primary_key=True, default=uuid.uuid4, editable=False)
name = models.CharField(max_length=120)
def __str__(self) -> str:
return self.name
class Session(models.Model):
"""A meditation/discourse session with a fixed number of seats."""
id = models.UUIDField(primary_key=True, default=uuid.uuid4, editable=False)
title = models.CharField(max_length=200)
starts_at = models.DateTimeField()
seat_limit = models.PositiveSmallIntegerField()
def __str__(self) -> str:
return f"{self.title} @ {self.starts_at:%Y-%m-%d %H:%M}"
class SeatBooking(models.Model):
"""Counted inventory — correctness comes from a row lock on the Session."""
id = models.UUIDField(primary_key=True, default=uuid.uuid4, editable=False)
session = models.ForeignKey(Session, on_delete=models.PROTECT, related_name="bookings")
guest = models.ForeignKey("accounts.User", on_delete=models.PROTECT, related_name="seat_bookings")
status = models.CharField(max_length=12, choices=BookingStatus.choices, default=BookingStatus.CONFIRMED)
created_at = models.DateTimeField(auto_now_add=True)
class Meta:
indexes = [models.Index(fields=["session", "status"])]
def __str__(self) -> str:
return f"seat {self.id} / {self.session_id}"
class RoomStay(models.Model):
"""Geometric inventory — correctness comes from the exclusion constraint below."""
id = models.UUIDField(primary_key=True, default=uuid.uuid4, editable=False)
room = models.ForeignKey(Room, on_delete=models.PROTECT, related_name="stays")
guest = models.ForeignKey("accounts.User", on_delete=models.PROTECT, related_name="stays")
starts_at = models.DateTimeField()
ends_at = models.DateTimeField()
status = models.CharField(max_length=12, choices=BookingStatus.choices, default=BookingStatus.PENDING)
created_at = models.DateTimeField(auto_now_add=True)
class Meta:
constraints = [
ExclusionConstraint(
name="no_overlapping_confirmed_stays",
expressions=[
("room", RangeOperators.EQUAL),
(
Func(
F("starts_at"),
F("ends_at"),
Value("[)"), # half-open: check-out == next check-in is NOT a clash
function="tstzrange",
output_field=DateTimeRangeField(),
),
RangeOperators.OVERLAPS,
),
],
condition=Q(status=BookingStatus.CONFIRMED),
),
]
def __str__(self) -> str:
return f"stay {self.id} / room {self.room_id}"
The SeatBooking half looks unremarkable on purpose — its correctness lives in the service, not the schema, because “no more than N of these” isn’t something a constraint expresses cheaply. The RoomStay half is the opposite: its correctness is the schema. Hold that thought; we build each one in turn.
Row locking — SELECT ... FOR UPDATE
A database transaction is a bundle of statements that either all take effect together or none do — and while it’s open, it can hold locks. SELECT ... FOR UPDATE reads a row and locks it, so any other transaction that tries to lock the same row blocks — it waits, parked, until the first transaction commits or rolls back. Serialization by waiting.
For fixed-count inventory like seats, you lock the thing that represents the scarce resource — the session row that carries the seat count — before you read the remaining capacity:
# ambarsthan/bookings/services.py
from django.db import transaction
from .models import BookingStatus, Session, SeatBooking
class SeatSoldOut(Exception):
"""Raised when the session is full — the caller maps it to HTTP 409."""
def reserve_seat(session_id, guest) -> SeatBooking:
with transaction.atomic():
session = (
Session.objects
.select_for_update() # lock this ONE session row
.get(pk=session_id)
)
taken = (
SeatBooking.objects
.filter(session=session, status=BookingStatus.CONFIRMED)
.count() # re-read AFTER we hold the lock
)
if taken >= session.seat_limit:
raise SeatSoldOut()
return SeatBooking.objects.create(
session=session, guest=guest, status=BookingStatus.CONFIRMED,
)
# lock releases on commit, when the atomic() block exits
The select_for_update() is the whole difference. Once User A holds the lock on that session row, User B’s identical request stops dead at its own select_for_update() and does not move until A commits. By the time B is allowed to read the count, A’s booking already exists, B sees forty of forty, and B is correctly told the session is full. The window is gone because only one transaction can be inside it at a time.
sequenceDiagram
participant A as User A
participant B as User B
participant DB as Database
A->>DB: begin and lock session row
B->>DB: begin and lock session row
Note over B: B blocks and waits
A->>DB: seat free, insert booking, commit
DB-->>B: lock released
B->>DB: re-read count shows full
DB-->>B: rejected, sold outNote what this costs: B waits. Under a spike, requests for the same session queue up on that one lock and are served one at a time. That’s the point — for the contended resource you want serialization. You keep the locked section tiny (lock, check, insert, commit) so the queue drains fast, and you scope the lock to a single session row so bookings for other sessions never touch it and run fully in parallel.
Why the re-read under the lock is not redundant
This is the part everyone squints at: if we already read the count, why does re-reading it after the lock give a different, correct answer? The answer is Postgres’s default isolation level, READ COMMITTED, and it’s worth being precise because the mental model most people carry is wrong.
Under READ COMMITTED, each statement gets its own snapshot of “everything committed so far” — not each transaction. B’s select_for_update() blocks on A’s lock and only wakes once A commits (A never updates the session row here — it just holds the lock — so there’s no new row version for B to re-check; B simply acquires the lock the instant A releases it). The crucial part comes next: B’s next statement, the count(), takes a fresh snapshot that now includes A’s just-committed SeatBooking. That’s the entire trick. The count() isn’t redundant with the one A ran; it runs later, on a newer snapshot, because the lock forced B to wait until after A was durable. Read the count before taking the lock and you’re back in the racing version — the lock is only useful if the decision-making read happens after you hold it.
If we needed the whole transaction to see one frozen snapshot instead — for a multi-statement invariant — we’d reach for REPEATABLE READ or SERIALIZABLE, but those turn write conflicts into could not serialize access errors you have to retry. For a single contended counter, a row lock under READ COMMITTED is simpler and cheaper, and it’s what we run.
When you don’t want to wait: NOWAIT, SKIP LOCKED, lock_timeout
Blocking is the right default for a seat everyone wants, but “park and wait indefinitely” is a foot-gun in a web request — a stuck lock holder becomes a pile-up of stuck gunicorn workers. Three knobs shape the waiting, and each has a real use here:
lock_timeoutbounds the wait. Set it per-transaction so a lock we can’t get in a few seconds fails loudly instead of holding a worker hostage. We setSET LOCAL lock_timeout = '3s'at the top of the atomic block; exceeding it raisesOperationalError(lock_not_available), which we translate to a “system busy, try again” rather than a 30-second hang.select_for_update(nowait=True)doesn’t wait at all — if the row is locked, it errors immediately. Use it when you’d rather instantly tell the user “someone’s booking that right now, refresh” than queue them.select_for_update(skip_locked=True)skips locked rows and moves to the next unlocked one. Wrong for a specific contended seat, but exactly right for “hand me any free room from this pool” or for draining a work queue — it’s how the Celery-adjacent job pickers avoid all fighting over the same head row.
from django.db import connection, transaction
def reserve_seat_bounded(session_id, guest) -> SeatBooking:
with transaction.atomic():
with connection.cursor() as cur:
cur.execute("SET LOCAL lock_timeout = '3s'") # don't hang a worker forever
session = Session.objects.select_for_update().get(pk=session_id)
... # same body as reserve_seat
Deadlocks and lock ordering
The moment a single request locks more than one row, deadlocks become possible: transaction 1 holds row X and wants Y, transaction 2 holds Y and wants X, and both wait forever until Postgres kills one with deadlock detected. Ambarsthan hits this on group bookings — one request reserving three specific rooms for a family. The fix is boring and total: always acquire locks in a deterministic order, so two overlapping requests can never form a cycle.
def reserve_rooms(room_ids, guest, starts_at, ends_at) -> list[RoomStay]:
with transaction.atomic():
# Deterministic lock order (by pk) makes a hold-and-wait cycle impossible.
rooms = list(
Room.objects
.select_for_update()
.filter(pk__in=room_ids)
.order_by("pk")
)
return [
RoomStay.objects.create(
room=room, guest=guest,
starts_at=starts_at, ends_at=ends_at,
status=BookingStatus.CONFIRMED,
)
for room in rooms
]
The order_by("pk") is load-bearing, not cosmetic: two requests each grabbing rooms {A, C} and {C, A} both lock in the order A, C, so one simply waits for the other instead of deadlocking. Any stable, agreed-upon order works as long as every code path uses the same one — which is itself an argument for funnelling all booking writes through this one service.
What it costs in wall-clock. The locked section is tiny — lock, count, insert, commit — and measures 2–4 ms on our box. When registration opened for a popular session and roughly 800 taps landed on one session inside ~3 seconds, they serialized on that single row lock and the queue drained in under 1.5 s end to end; p99 wait to acquire the lock sat around 1.1 s, and not a single seat oversold. Because the lock is scoped to one session row, the other forty sessions booking at the same moment never contended at all.
Exclusion constraints — for overlapping time ranges
Seats are a counting problem. Stays are a geometry problem. A room booked from the 4th to the 7th conflicts with one from the 6th to the 9th because the intervals overlap — and “overlap” is not something a plain unique constraint can express. A unique index stops two identical rows; it has no idea that [4,7) and [6,9) collide.
Postgres has a purpose-built tool for exactly this: the exclusion constraint. A UNIQUE constraint says no two rows may be equal on some columns. An EXCLUDE constraint generalizes that to any operator — including &&, the range-overlap operator. You declare, at the schema level, “no two confirmed bookings may exist where the room is the same and the time ranges overlap,” and the database enforces it on every insert, forever, no matter which application, cron job, or hand-run SQL touches the table.
CREATE EXTENSION IF NOT EXISTS btree_gist;
ALTER TABLE bookings
ADD CONSTRAINT no_overlapping_confirmed_stays
EXCLUDE USING gist (
room_id WITH =,
tstzrange(starts_at, ends_at) WITH &&
)
WHERE (status = 'confirmed');
A few things worth unpacking, because each earns its keep:
tstzrange(starts_at, ends_at)turns two timestamp columns into a single range value. By default it’s half-open —[start, end)— so a stay ending at noon and another starting at noon do not overlap. That boundary convention is a decision you want made once, in the schema, not re-derived in three different services.&&is “these ranges overlap.”WITH =onroom_idmeans the rule only applies within the same room. Together: same room, overlapping time = rejected.USING gistis the index type that can answer overlap questions. Andbtree_gistis the extension that lets a GiST index also handle the plain equality onroom_id— without it, you can’t mix=and&&in one constraint.WHERE (status = 'confirmed')makes it a partial constraint: cancelled or pending rows are exempt, so a cancelled booking never blocks a real one.
That SQL is what Postgres runs, but I never hand-wrote it — the ExclusionConstraint in RoomStay.Meta generates it. The one thing Django can’t infer is that GiST needs the btree_gist extension to mix the = on room_id with the && on the range, so the migration installs it first. BtreeGistExtension (and the RangeOperators mapping to &&) live in django.contrib.postgres:
# ambarsthan/bookings/migrations/0007_no_overlapping_stays.py
from django.contrib.postgres.constraints import ExclusionConstraint
from django.contrib.postgres.fields import DateTimeRangeField, RangeOperators
from django.contrib.postgres.operations import BtreeGistExtension
from django.db import migrations, models
from django.db.models import F, Func, Q, Value
class Migration(migrations.Migration):
dependencies = [("bookings", "0006_roomstay")]
operations = [
BtreeGistExtension(), # emits CREATE EXTENSION IF NOT EXISTS btree_gist
migrations.AddConstraint(
model_name="roomstay",
constraint=ExclusionConstraint(
name="no_overlapping_confirmed_stays",
expressions=[
("room", RangeOperators.EQUAL),
(
Func(
F("starts_at"), F("ends_at"), Value("[)"),
function="tstzrange",
output_field=DateTimeRangeField(),
),
RangeOperators.OVERLAPS,
),
],
condition=Q(status="confirmed"),
),
),
]
One production note before you run this: BtreeGistExtension needs a role with CREATE on the database (superuser or an owner with the privilege). On managed Postgres — RDS, Cloud SQL — btree_gist is on the allow-list, so CREATE EXTENSION works under the regular migration role, but confirm that before the deploy rather than during it. And AddConstraint builds the GiST index while holding an ACCESS EXCLUSIVE lock on the table; on an empty or small roomstay it’s instant, but on a large existing table you’d want to create the index CONCURRENTLY first and then attach it, or take the write-lock in a maintenance window.
The beautiful part is that there is no check-then-act to race. Two overlapping inserts arriving at the same instant both go to commit; Postgres serializes them at the constraint and lets exactly one win. The loser gets a constraint-violation error — which Django raises as IntegrityError — so the service catches exactly that constraint by name and returns a clean 409, never leaking a 500:
# ambarsthan/bookings/services.py (continued)
from django.db import IntegrityError, transaction
class SlotJustTaken(Exception):
"""The exclusion constraint rejected an overlapping stay."""
def reserve_stay(room_id, guest, starts_at, ends_at) -> RoomStay:
try:
with transaction.atomic():
return RoomStay.objects.create(
room_id=room_id, guest=guest,
starts_at=starts_at, ends_at=ends_at,
status=BookingStatus.CONFIRMED,
)
except IntegrityError as exc:
# Only the overlap constraint means "someone beat you to it".
# Any other IntegrityError (FK, not-null) is a real bug — re-raise it.
if "no_overlapping_confirmed_stays" in str(exc):
raise SlotJustTaken() from exc
raise
# ambarsthan/bookings/views.py
from rest_framework import status
from rest_framework.response import Response
from rest_framework.views import APIView
from .services import SlotJustTaken, reserve_stay
class StayBookingView(APIView):
def post(self, request):
data = request.data
try:
stay = reserve_stay(
room_id=data["room_id"], guest=request.user,
starts_at=data["starts_at"], ends_at=data["ends_at"],
)
except SlotJustTaken:
return Response(
{"detail": "just_taken",
"message": "That slot just went. Here are the next open dates."},
status=status.HTTP_409_CONFLICT,
)
return Response(StaySerializer(stay).data, status=status.HTTP_201_CREATED)
Matching on the constraint name rather than swallowing every IntegrityError matters: a null-violation or a broken foreign key is a genuine bug, and turning it into a friendly 409 would hide it. Only the named overlap collision becomes “just taken.”
There’s a free bonus in that constraint. The GiST index Postgres built to enforce it is the same index that makes availability lookups fast — “is room 12 free from the 4th to the 7th?” is the identical overlap question. Here’s the query that scans the calendar for a clash, on a roomstay table with ~182k historical rows, before the constraint existed:
EXPLAIN (ANALYZE, BUFFERS)
SELECT id FROM bookings_roomstay
WHERE room_id = 'a1f0…'::uuid
AND status = 'confirmed'
AND tstzrange(starts_at, ends_at) && tstzrange('2026-04-04', '2026-04-07', '[)');
Seq Scan on bookings_roomstay (cost=0.00..6218.54 rows=1 width=16)
(actual time=214.883..418.221 rows=1 loops=1)
Filter: ((status = 'confirmed'::text)
AND (tstzrange(starts_at, ends_at, '[)') && '["2026-04-04","2026-04-07")'::tstzrange)
AND (room_id = 'a1f0…'::uuid))
Rows Removed by Filter: 182443
Buffers: shared hit=192 read=2246
Planning Time: 0.141 ms
Execution Time: 418.402 ms
A full sequential scan, 182,443 rows thrown away by the filter, 2,246 blocks read off disk, 418 ms. After the exclusion constraint added its partial GiST index, the same query rides it — the status = 'confirmed' predicate is covered by the partial index condition, so it doesn’t even appear in the filter:
Index Scan using no_overlapping_confirmed_stays on bookings_roomstay
(cost=0.28..8.30 rows=1 width=16) (actual time=0.058..0.061 rows=1 loops=1)
Index Cond: ((room_id = 'a1f0…'::uuid)
AND (tstzrange(starts_at, ends_at, '[)') && '["2026-04-04","2026-04-07")'::tstzrange))
Buffers: shared hit=5
Planning Time: 0.132 ms
Execution Time: 0.094 ms
Five buffers, all cache hits, 0.09 ms — roughly 4,000× faster, and it’s a side effect of a constraint we added for correctness. The mechanism that stops the double-booking also answers “what’s available” for free.
Why the database, and not a Redis lock
The obvious alternative — the one a lot of teams reach for — is an application-layer lock. Spin up a mutex (a “mutual exclusion” token only one worker can hold) in Redis, grab it before booking, release it after. It works. I chose not to build correctness on it, for reasons that are about where the truth lives.
A Redis lock is advisory: it only protects you if every single code path remembers to take it. The moment someone writes a bulk import, a refund-and-rebook admin action, or a one-off data fix in a Django shell — and forgets the lock, because it’s just a convention in the app — the invariant is silently gone. Worse, distributed locks have genuinely hard failure modes: the lock’s TTL expires mid-operation, the holder was paused by GC or a slow disk, and now two workers believe they hold it. Correctness that depends on a clock and a network is not correctness; it’s a probability.
The exclusion constraint and the row lock have the opposite property. They are enforced by the one component that every write path must go through no matter what — the database. There is no code path that reaches the table without passing the constraint. You cannot forget it, you cannot bypass it from a shell, and it does not care whether the request came from the API, a Celery task, or a DBA’s psql session. The invariant isn’t guarded by discipline; it’s true by construction.
The outcome
Once the invariants moved into the schema, double-booking stopped being a bug we chased and became a state the database refuses to represent. During registration spikes we now watch the constraint do exactly what it should: a handful of losing inserts bounce off it per popular session, we translate them into “that slot just went — here are the next available ones,” and the user re-picks. No overselling, no reconciliation job cleaning up phantom bookings the morning after, no support tickets about two families assigned the same room. The application code got simpler, not more complex — it stopped trying to be careful, because it no longer had to be. The database is careful for it.
The lesson
For correctness under concurrency, push the invariant down to the lowest layer that every writer must cross — and in a system with a relational database, that layer is the database. The application cannot win a race the database was purpose-built to arbitrate: your if statement and your Redis lock are running in userspace, milliseconds and network hops away from the truth, while SELECT ... FOR UPDATE and EXCLUDE are the truth. Pick the right tool for the shape of the scarcity — row locks for counted inventory, exclusion constraints for overlapping ranges — and then stop guarding the rule in code you can forget to call. Make the illegal state unrepresentable, and the whole class of bug evaporates.
Correctness is now enforced by the schema. But Ambarsthan and the ERP that shares this backend have a different kind of question waiting — not “can two people take the same seat,” but “who is even allowed to confirm a booking, or open a session, or refund a stay.” Next, we let the organization define that for itself.