Contents
The first time I saw it, a founder saved their startup profile and Elasticsearch fell over.
Not literally — but close enough that the web request timed out and the search index spent the next several minutes catching up. One save() in Django had quietly fanned out into thousands of individual index operations. I’d wired the search layer up the textbook way, the way every django-elasticsearch-dsl tutorial shows you, and it worked beautifully right up until the data model got interesting. This is the story of how I stopped reindexing per record and started reindexing in batches — and the pitfalls that pattern hides.
This was on a startup-fundraising platform I ran for the better part of four years: startups build a profile and get matched against a large curated investor database, ranked by a weighted score. Search and match scoring both live in Elasticsearch — the setup where reindex storms are born.
Why per-record reindexing bites at scale
django-elasticsearch-dsl ships a RealTimeSignalProcessor. You register a document, and every time the underlying model saves, a post_save signal fires and reindexes that document synchronously, inside the same request. For a blog with a Post model, this is perfect. You never think about it.
The trouble starts when your search document is denormalized — when one Elasticsearch document depends on many rows in Postgres. On this platform, an investor-contact document carried its match data inline: the weighted score against each startup, the sector overlap, the stage fit, the geography match. Search was match scoring, so the score had to be in the index.
Now trace what happens when a startup edits its profile. The match score against that startup changes for every contact in the database. So the app recomputes thousands of MatchScore rows. Each of those rows is wired to reindex its parent contact document. And because a single contact can be touched several times in one recompute pass — sector, then stage, then region — the same document gets reindexed again and again.
One user action. Thousands of Elasticsearch writes, most of them redundant, all of them synchronous. That’s a reindex storm, and it has three distinct costs:
- Latency — the write blocks the request that triggered it.
- Load — Elasticsearch spends its time servicing a flood of single-document writes instead of serving search.
- Waste — reindexing the same document forty times to reach the same final state.
The conventional fix, and why I didn’t stop there
The obvious move is to get the reindex off the request thread. django-elasticsearch-dsl even hands you a CelerySignalProcessor for exactly this — instead of reindexing inline, it fires a Celery task per instance. Problem solved, right?
Only partly. Async per-record reindexing fixes the latency problem and does nothing for the other two. You’ve turned thousands of synchronous writes into thousands of Celery tasks, which then become thousands of Elasticsearch writes a beat later. The same document still gets reindexed forty times; you’ve just moved the storm one hop downstream and added queue pressure on top.
So I asked a different question — not “how do I make this async?” but “how fresh does this index actually need to be?” And the honest answer, for this product, was: not very. Nobody is watching match scores update in real time. A recompute that lands in search within a few minutes is completely indistinguishable, to a user, from one that lands instantly. That slack — the gap between real-time and good enough — is the thing I got to trade away. Freshness I didn’t need bought me stability I very much did.
That reframe is the whole post. The batching mechanism below is easy. Noticing that the freshness requirement was softer than the framework’s default assumed is the part that actually mattered, and it’s the kind of call you only make if you own the product, not just the ticket.
A durable sync queue instead of a signal
The design: signals stop reindexing anything. They enqueue. A row lands in a PendingElasticsearchSync table saying “this object is dirty.” A cron’d management command drains that table every 15 minutes, in batches of 500, with a single Elasticsearch bulk call. That’s it.
The queue model does the heavy lifting, and the unique constraint is the point — it’s what collapses a hot record’s forty writes into one:
from django.db import models
from django.contrib.contenttypes.models import ContentType
class PendingElasticsearchSync(models.Model):
class Action(models.TextChoices):
INDEX = "index"
DELETE = "delete"
content_type = models.ForeignKey(ContentType, on_delete=models.CASCADE)
object_id = models.PositiveBigIntegerField()
action = models.CharField(max_length=6, choices=Action.choices, default=Action.INDEX)
dirtied_at = models.DateTimeField(default=timezone.now)
class Meta:
constraints = [
models.UniqueConstraint(
fields=["content_type", "object_id"],
name="uniq_pending_es_sync_per_object",
)
]
indexes = [models.Index(fields=["dirtied_at"])]
Enqueuing replaces the reindex in the signal handler. update_or_create on the unique key means a record dirtied a hundred times between drains is still exactly one row waiting to be processed:
def enqueue_sync(instance, action=PendingElasticsearchSync.Action.INDEX):
PendingElasticsearchSync.objects.update_or_create(
content_type=ContentType.objects.get_for_model(instance),
object_id=instance.pk,
defaults={"action": action, "dirtied_at": timezone.now()},
)
The drain reads a batch, builds one bulk payload, ships it, and clears what it processed:
def handle(self, *args, **options):
claim_started = timezone.now()
batch = list(PendingElasticsearchSync.objects.order_by("dirtied_at")[:500])
if not batch:
return
helpers.bulk(es_client, build_bulk_actions(batch))
# Delete only rows we actually processed — and only if they weren't
# re-dirtied while we were building the payload (see pitfall #3).
ids = [row.pk for row in batch]
PendingElasticsearchSync.objects.filter(
pk__in=ids, dirtied_at__lte=claim_started
).delete()
Run that command from cron every 15 minutes and the arithmetic is stark: 4 drains an hour, at most 500 documents each, one bulk request per drain. Per the platform’s own docs, this took Elasticsearch write volume from potentially thousands of operations per minute down to 4 batch operations per hour. The index is at most 15 minutes stale, which — for match scores nobody watches live — is free.
The pitfalls, because there are always pitfalls
This pattern is simple to draw and easy to get subtly wrong. Four things bit me or nearly did.
1. Deletes need a tombstone. When a contact is deleted, the row is gone from Postgres — but its document is still in Elasticsearch, and the queue can’t reindex a record that no longer exists. That’s why the action lives on the queue row. A post_delete enqueues action=DELETE, so the drain knows to issue a delete op instead of trying to load and reindex a ghost. Build the bulk actions with that branch explicit:
def build_bulk_actions(rows):
for row in rows:
if row.action == PendingElasticsearchSync.Action.DELETE:
yield {"_op_type": "delete", "_index": INDEX, "_id": row.object_id}
continue
instance = row.content_type.model_class().objects.filter(pk=row.object_id).first()
if instance is None: # deleted between enqueue and drain — delete anyway
yield {"_op_type": "delete", "_index": INDEX, "_id": row.object_id}
continue
yield {"_op_type": "index", "_index": INDEX, "_id": instance.pk,
"_source": ContactDocument().prepare(instance)}
Side note: that per-row query is an N+1. At 500 rows it’s tolerable, but the clean version groups the batch by content_type and fetches each model’s instances in one filter(pk__in=...) — worth doing before the queue grows.
2. The drain has to be idempotent. It’s a cron job; it will crash mid-batch someday and run again. Because the delete happens after a successful bulk call, a crash before the delete just means the same rows reindex next cycle — and reindexing the same document twice is harmless. At-least-once delivery is exactly what you want. Never delete the queue rows before you’ve confirmed the write landed.
3. Mind the write that lands mid-drain. This is the sneaky one. The drain reads a row, then builds the payload, then deletes the row. If a fresh update to that same object lands after the read but before the delete, a naive “delete everything I claimed” wipes the signal and the document goes stale until something unrelated touches it again. The fix is the dirtied_at__lte=claim_started filter above: capture a timestamp before reading, and delete only rows that haven’t been re-dirtied since. Anything touched mid-drain keeps a newer dirtied_at, survives the delete, and drains next cycle. Cheap optimistic concurrency, no locks.
4. Don’t route a full rebuild through the queue. A cold index or a mapping change needs every document reindexed. Feeding millions of rows through a 500-every-15-minutes drain would take days. Keep a separate, boring management command that bulk-reindexes from scratch in large chunks. The queue is for incremental drift, not for standing the index up.
And one operational note: your health metric changes. With inline reindexing, a broken index is loud. With a queue, it’s silent — the app happily enqueues while nothing drains. Queue depth is your new alarm. If pending rows climb without falling, the drain has stalled, and you want to know before search goes an hour stale, not after a user reports it.
When not to reach for this
If a user edits their own record and must see it reflected in search on the very next screen, a 15-minute queue is the wrong tool for that path. The right shape there is a hybrid: keep a fast, direct reindex for the handful of self-service edits where the user is staring at the result, and batch only the fan-out — the bulk recomputes nobody is watching. Batching is a freshness-for-stability trade, and you should only make it where you can actually spare the freshness.
That was the real lesson. The queue table, the unique constraint, the bulk drain — that’s an afternoon of code. The engineering was deciding, on behalf of the product, that match-score search could lag a few minutes and nobody would care. Get that judgment right and thousands of writes a minute quietly become four an hour. Get it wrong and you’ve traded away something users needed. The mechanism is the easy half.