Contents
The terabytes had finished migrating. Both object stores had been in sync until the atomic cutover, the CDN was serving from Bunny, and for the first time in a long while nothing about our media stack was on fire. Thirty thousand users, roughly sixty percent of them paying, streaming courses and audios and long-form videos in Hindi and English, mostly on mid-range Android phones on Indian mobile data. The platform was stable.
And people still could not find anything.
The problem hiding behind a working search box
The search box worked. That was the trap. Type a word, get results, no errors in the logs. But the results were wrong in a way dashboards do not catch. A user looking for a talk on inner peace would type shanti, or शांति, or mann ki shanti, or simply peace, and get four different, mostly empty result sets for what was obviously one intent.
Our search was keyword search — the classic approach where the database matches the literal words you typed against the literal words in the content. Postgres does this well for English through its full-text engine: it reduces text to a tsvector of normalized lexemes, reduces the query to a tsquery, and matches the two. You type meditation, it finds rows containing meditation, and with a stemming dictionary it also finds meditations and meditating. Fast, predictable, and it had been fine at two thousand users.
Hindi broke it on three fronts at once, and the fastest way to show it is to run the query that was quietly failing in production:
-- Content title stored in Devanagari.
-- User typed the same word, romanized, on a phone keyboard.
SELECT id, title
FROM content_item
WHERE to_tsvector('simple', title) @@ plainto_tsquery('simple', 'dhyan');
id | title
----+-------
(0 rows)
The row exists. Its title is ध्यान और मन की शांति. The query means exactly that. Postgres returns nothing, and — this is the trap — it returns nothing without an error. The dashboard sees a successful query with zero results, indistinguishable from a genuine miss. Three separate mechanisms conspire here.
First, morphology — the way a language changes the shape of its words. Hindi inflects far more heavily than English. A single root verb surfaces as a dozen written forms depending on tense, gender, and politeness (करता, करती, करते, किया, करूँगा…). Postgres ships no stemming dictionary for Hindi, so to_tsvector('simple', …) treats every one of those forms as an unrelated token. Match one, miss the other eleven.
Second, transliteration — writing one language in another’s script. Half our users typed Hindi in Roman letters — dhyan instead of ध्यान — and there is no single correct spelling. dhyan, dhyaan, dyan share not one byte with the Devanagari string in the column, so the @@ match above can never fire. A trigram index (pg_trgm) rescues typo variants of the same script, but it too compares raw characters, so Roman-vs-Devanagari stays at zero overlap.
Third, cross-language intent. The library was bilingual. Someone typing peace genuinely wanted that Hindi talk on शांति, and no amount of stemming will ever connect two words that share not a single character.
You cannot stem your way out of this. The failure is not in the spelling of words; it is that the search understands words but not meaning.
Crash course: from matching words to matching meaning
The fix is a different kind of search entirely, so it is worth building the idea up from nothing.
Semantic search asks a different question than keyword search. Keyword search asks “which documents contain these exact words?” Semantic search asks “which documents mean roughly the same thing as this query?” To answer the second question, a computer needs some way to represent meaning as something it can compare.
That representation is an embedding. An embedding is just a list of numbers — a vector — that a machine-learning model produces for a piece of text. The trick, and it really is the whole trick, is that the model is trained so that texts with similar meanings get vectors that sit close together, and texts with unrelated meanings get vectors that sit far apart. शांति, shanti, and peace all land in nearly the same neighborhood. meditation lands in an adjacent one. invoice lands on the other side of the map.
A useful mental model: imagine every sentence gets dropped as a pin onto an enormous map, arranged so that things about the same topic cluster together regardless of the words or script used. Searching becomes “drop a pin for the query, then hand back whatever pins are nearest.” That nearness is measured with cosine distance — a number that is small when two vectors point in the same direction and large when they do not. The direction encodes the meaning; the distance measures the disagreement.
There is one more piece. Which model draws the map matters enormously. A model trained mostly on English will crush Hindi into a vague blur. We used an Indic-SBERT model — a sentence-transformer variant (l3cube-pune/indic-sentence-bert-nli) trained on Indian languages, so it maps Hindi and Hinglish and English into the same shared space with real precision. Feed it a sentence, get back a vector of exactly 768 numbers — that fixed width is a property of the model, and it becomes a hard constraint on everything downstream: the Postgres column, the index, and the query vector all have to be 768-dimensional or the math simply does not line up. That is its entire job.
Where do the vectors live? The decision that shaped everything
Now the architecture question, and it is the one I want a future CTO reading this to sit with, because the tempting answer is the wrong one.
To do semantic search you need to store a vector for every searchable item and, at query time, find the stored vectors nearest to the query’s vector — fast, over your whole library. The industry’s default reflex is to reach for a dedicated vector database: a separate system, purpose-built for exactly this, with clever approximate-nearest-neighbor indexes and horizontal scaling.
I did not want a new database.
Here is the honest ledger. A dedicated vector store buys you best-in-class similarity search at very large scale — think hundreds of millions of vectors and up. What it costs you is a second source of truth. Now your content lives in Postgres and its vectors live somewhere else, and those two stores must be kept in sync forever. Every publish, edit, unpublish, and soft-delete has to fan out to both. When they drift — and they drift — your search quietly lies. On top of that you inherit a second thing to back up, monitor, secure, and reason about at 2 a.m.
The alternative is pgvector: an extension that teaches the Postgres we were already running to store vectors natively and compare them. It adds a real vector column type and a distance operator, so a nearest-neighbor search becomes an ordinary ORDER BY ... LIMIT in SQL — inside the same transaction, right next to the content it describes.
That last part is the quiet superpower. Our content is not just text; it has state. Is this course published? Which language? Is it soft-deleted? Does this user’s plan include it? With pgvector, the similarity search is a plain column in the same table, so all of that filtering happens in one query, against one consistent snapshot of the data, with the joins we already had. No sync job. No drift. No second 2 a.m. system.
The rule I applied is the one I have applied through this entire series: add capability inside the system you already operate before you add a new system. A dedicated vector database is the right answer when you have genuinely outgrown Postgres — billions of vectors, similarity search as your primary and highest-volume workload. We had tens of thousands of items and search as one feature among many. Reaching for a distributed vector store at that scale would have been taking a second mortgage to add a room we could build in the house we already owned.
So: stay in Postgres. Until we truly outgrow it, and we had not.
The one piece that does not belong in Django
There was exactly one part of this that could not live inside the web app: the model itself.
The Indic-SBERT model is a chunk of PyTorch weights that wants to load into memory once and stay warm. Our Django API runs as many worker processes behind gunicorn, each a copy of the app. Loading a heavy neural model into every one of those workers would balloon memory and slow every deploy, and most of those workers never touch search.
So the model runs as an embedding sidecar — a small, separate service whose only job is: receive text, return a vector. We wrote it as a lean FastAPI app that loads the model once at startup and exposes a single endpoint. Django never imports PyTorch. When it needs a vector, it makes one HTTP call to the sidecar. That is the whole contract.
# embed_service/main.py — the entire sidecar, minus logging.
from fastapi import FastAPI
from pydantic import BaseModel
from sentence_transformers import SentenceTransformer
MODEL_NAME = "l3cube-pune/indic-sentence-bert-nli" # 768-dim, multilingual Indic
EXPECTED_DIM = 768
# Loaded ONCE, at process start, into this one service — not into every
# gunicorn worker of the Django app.
model = SentenceTransformer(MODEL_NAME)
app = FastAPI()
class EmbedRequest(BaseModel):
texts: list[str]
class EmbedResponse(BaseModel):
model: str
dim: int
vectors: list[list[float]]
@app.post("/embed", response_model=EmbedResponse)
def embed(req: EmbedRequest) -> EmbedResponse:
"""Encode a batch of texts to L2-normalized 768-dim vectors.
normalize_embeddings=True is deliberate: once vectors are unit-length,
cosine distance and inner product agree, so the cosine index in Postgres
ranks exactly the way this model was trained to be compared.
"""
vectors = model.encode(
req.texts,
normalize_embeddings=True,
batch_size=32,
)
assert vectors.shape[1] == EXPECTED_DIM # fail loud if the model is swapped
return EmbedResponse(model=MODEL_NAME, dim=vectors.shape[1], vectors=vectors.tolist())
“Sidecar” is the operative word: it rides alongside the main app, scoped to one narrow job, scaled and deployed on its own schedule. If we swap the embedding model later, we redeploy one small service and re-embed; Django does not even notice. It is the same instinct as the separate transcoding service from earlier chapters — keep the heavy, specialized, model-shaped work out of the request path and behind a clean boundary.
sequenceDiagram
participant U as User
participant D as Django
participant S as SBERT sidecar
participant P as Postgres
U->>D: search query text
D->>S: embed this text
S->>D: 768 dim vector
D->>P: keyword plus vector query
P->>D: merged ranked rows
D->>U: resultsHybrid: keeping the best of both worlds
Semantic search has one weakness worth naming: it is fuzzy by design. Ask it for an exact title, a proper noun, a course code, and its instinct to generalize can bury the exact match under things that are merely related. Keyword search, the thing we were replacing, is precise about exactly those cases.
So we did not replace it. We ran both and merged them — a hybrid search.
Every query fans out two ways at once. The keyword arm uses Postgres full-text search plus trigram matching (which compares little three-character slices of words, so it tolerates typos and spelling drift — a real help with transliteration). The semantic arm embeds the query through the sidecar and asks pgvector for the nearest content vectors. Then we fuse the two ranked lists.
For the fusion we used Reciprocal Rank Fusion, which is far simpler than its name. Each arm produces a ranked list. An item’s score is one divided by its rank in each list, summed across both. Something ranked first in either list scores highly; something that shows up respectably in both lists floats to the very top. It needs no tuning of “how much do we trust vectors versus keywords,” which is exactly the knob you do not want to be hand-adjusting in production. Rank position is the only input.
flowchart LR
Q[User query<br/>Hindi or Hinglish] --> KW[Keyword arm<br/>full-text plus trigram]
Q --> SB[Indic-SBERT sidecar<br/>text to vector]
SB --> PV[pgvector<br/>nearest by cosine distance]
KW --> M[Reciprocal rank fusion]
PV --> M
M --> R[Merged ranked results]The schema, in Django, not raw SQL
One decision shaped the table layout: we embed at passage granularity, not whole items. A ninety-minute discourse is one content_item, but its transcript splits into a few dozen searchable content_chunk rows. A user asking about one idea inside a long talk should land on the passage that discusses it, then resolve up to its parent item. Tens of thousands of items became roughly 1.2 million chunks — which, as we will see, is exactly the scale at which the choice of index stops being academic.
The pgvector-django package gives us a real VectorField, so the column lives in the model where the rest of the schema does:
# content/models.py
from django.db import models
from pgvector.django import VectorField, HnswIndex
EMBEDDING_DIM = 768 # must equal the Indic-SBERT model's output width
class ContentChunk(models.Model):
item = models.ForeignKey(
"content.ContentItem", on_delete=models.CASCADE, related_name="chunks"
)
lang = models.CharField(max_length=8, db_index=True) # 'hi', 'en', ...
passage = models.TextField()
# 768 is not arbitrary: it is the sidecar model's fixed output width.
embedding = VectorField(dimensions=EMBEDDING_DIM, null=True)
# Guards against re-embedding text that has not changed (see Celery task).
embedding_source_hash = models.CharField(max_length=64, blank=True)
class Meta:
indexes = [
HnswIndex(
name="content_chunk_embedding_hnsw",
fields=["embedding"],
m=16, # graph out-degree; build/recall knob
ef_construction=64, # candidate list during build
opclasses=["vector_cosine_ops"], # cosine, matching the model
),
]
The migration that ships it enables the extension and builds the index in the same file, so a fresh database bootstraps itself:
# content/migrations/0042_content_chunk_embedding.py
from django.db import migrations
from pgvector.django import VectorExtension, VectorField, HnswIndex
class Migration(migrations.Migration):
dependencies = [("content", "0041_prior")]
operations = [
VectorExtension(), # CREATE EXTENSION IF NOT EXISTS vector
migrations.AddField(
model_name="contentchunk",
name="embedding",
field=VectorField(dimensions=768, null=True),
),
migrations.AddIndex(
model_name="contentchunk",
index=HnswIndex(
name="content_chunk_embedding_hnsw",
fields=["embedding"],
m=16,
ef_construction=64,
opclasses=["vector_cosine_ops"],
),
),
]
m and ef_construction are build-time knobs — higher values build a denser graph that recalls more true neighbors at the cost of build time and RAM; 16/64 is pgvector’s sensible default and it held up. The one knob you set at query time is hnsw.ef_search: how many candidates the traversal keeps in flight. It trades latency for recall, per-connection, without rebuilding anything:
SET hnsw.ef_search = 100; -- default 40; we raised it until recall plateaued
(HNSW is one of two index families here. The alternative, IVFFlat, partitions vectors into lists buckets and probes probes of them at query time — cheaper to build, but you must choose lists ≈ rows/1000 up front and it degrades if the data shifts. HNSW has no such upfront sizing, which is why it won for a library that grows continuously.)
The pipeline: embed on write, embed on read
Two moments produce a vector, and they must use the same model or the geometry is meaningless.
On write, whenever a chunk’s text is created or edited, Django enqueues a Celery task rather than calling the sidecar inline — encoding is model work, not request work, and it belongs off the web path on its own queue:
# app/celery.py
task_routes = {
"content.tasks.embed_chunk": {"queue": "embeddings"},
"payments.tasks.*": {"queue": "payments"}, # never waits behind embeddings
}
# content/tasks.py
import hashlib
import requests
from celery import shared_task
from django.conf import settings
from .models import ContentChunk
@shared_task(bind=True, acks_late=True, max_retries=3, default_retry_delay=10)
def embed_chunk(self, chunk_id: int) -> None:
"""Fetch a 768-dim vector from the sidecar and store it on the chunk.
acks_late keeps the message on the broker until the write commits, so a
worker crash mid-encode re-runs rather than silently dropping the vector.
"""
chunk = ContentChunk.objects.get(pk=chunk_id)
source_hash = hashlib.sha256(chunk.passage.encode("utf-8")).hexdigest()
if chunk.embedding is not None and chunk.embedding_source_hash == source_hash:
return # text unchanged since last embed — skip the model call
try:
resp = requests.post(
f"{settings.EMBED_SERVICE_URL}/embed",
json={"texts": [chunk.passage]},
timeout=10,
)
resp.raise_for_status()
except requests.RequestException as exc:
raise self.retry(exc=exc)
vector = resp.json()["vectors"][0]
# Targeted update: no full-model save, no signal storms, no lost concurrent edits.
ContentChunk.objects.filter(pk=chunk_id).update(
embedding=vector, embedding_source_hash=source_hash
)
The embeddings queue runs its own workers, so a burst of re-embeds after a bulk transcript import cannot stall a payment callback:
celery -A app worker -Q embeddings -c 2 --prefetch-multiplier 1
--prefetch-multiplier 1 matters here: each embed task is seconds of GPU-bound work, so we do not want a worker greedily reserving a queue of them while another sits idle.
On read, the same sidecar embeds the incoming query, and the KNN search is an ordinary Django queryset — CosineDistance in the order_by is what the <=> operator becomes in the ORM:
# content/search.py
from pgvector.django import CosineDistance
from .models import ContentChunk
def semantic_arm(query_vector: list[float], langs: list[str], k: int = 20):
"""Nearest-by-meaning chunks, filtered to the user's languages, in ONE query."""
return (
ContentChunk.objects
.filter(item__is_published=True, lang__in=langs) # plain state filters
.select_related("item") # no N+1 resolving parents
.order_by(CosineDistance("embedding", query_vector)) # HNSW satisfies this
[:k]
)
Filtering, joining, and semantic ranking in a single statement, against one consistent snapshot, with the model’s weight quarantined in a sidecar. Nothing to keep in sync.
The number that justified the whole exercise
Before the HNSW index existed, that ORDER BY … <=> had to compute cosine distance against every stored vector. EXPLAIN (ANALYZE, BUFFERS) on the 1.2M-chunk table is unambiguous about what that costs:
Limit (cost=98421.33..98421.38 rows=20 width=48)
(actual time=31284.512..31284.520 rows=20 loops=1)
Buffers: shared hit=1088 read=241902
-> Sort (cost=98421.33..99498.10 rows=430708 width=48)
(actual time=31284.509..31284.513 rows=20 loops=1)
Sort Key: (embedding <=> '[0.021,-0.088,...]'::vector)
Sort Method: top-N heapsort Memory: 46kB
Buffers: shared hit=1088 read=241902
-> Seq Scan on content_chunk
(cost=0.00..86971.85 rows=430708 width=48)
(actual time=0.041..30913.677 rows=431204 loops=1)
Filter: (lang = 'hi'::text)
Rows Removed by Filter: 771330
Buffers: shared hit=1088 read=241902
Planning Time: 0.194 ms
Execution Time: 31284.573 ms
A 31-second sequential scan, reading nearly a quarter-million buffers off disk and throwing away 771,330 rows after computing a full distance for each. Unusable — this is the “fine for a demo” full scan the earlier chapters warned about, at production scale.
With the HNSW index and hnsw.ef_search = 100, the same query stops scanning and starts traversing the graph:
Limit (cost=402.55..416.88 rows=20 width=48)
(actual time=182.334..204.116 rows=20 loops=1)
Buffers: shared hit=612 read=1874
-> Index Scan using content_chunk_embedding_hnsw on content_chunk
(cost=402.55..31240.90 rows=430708 width=48)
(actual time=182.330..204.098 rows=20 loops=1)
Order By: (embedding <=> '[0.021,-0.088,...]'::vector)
Filter: (lang = 'hi'::text)
Rows Removed by Filter: 143
Buffers: shared hit=612 read=1874
Planning Time: 0.211 ms
Execution Time: 204.181 ms
31,284 ms to 204 ms — a ~150× drop, and the buffer count falls from 242,990 to 2,486 because the traversal never touches most of the table. One caveat a staff reader should catch: the Filter … Rows Removed by Filter: 143 is applied after the ANN traversal returns candidates, so an aggressive language filter can under-fill the LIMIT. We over-fetch (ask for k * 4 candidates, then trim) and lean on recent pgvector’s iterative index scans to backfill; for our language mix it was never a problem in practice, but it is the sharp edge of HNSW-plus-filter and worth knowing before you ship it.
The outcome
Search stopped being a thing users worked around. shanti, शांति, mann ki shanti, and peace now converge on the same shelf of content, because in the model’s shared space they were always the same idea — we had simply never asked the question that way. Hinglish queries, the ones that had returned empty pages, started returning the obvious right answer. On a hand-labelled set of ~200 real queries pulled from the logs, top-3 relevance went from 0.41 to 0.87 while the end-to-end query — sidecar embed plus fused search — settled around 240 ms at the 95th percentile, most of it the model call, not Postgres. And because it was all still Postgres, it filtered by plan and language and publish-state for free, and it rode our existing backups and monitoring without a single new runbook.
We shipped a genuinely modern search capability without adding a single new database to operate. One extension, one small sidecar, one fused query.
The lesson, and the whole journey
The transferable lesson is the smallest sentence in this post: add capability inside the system you already operate before you add a new system. The exciting-sounding tool is a liability until you have proven the boring one cannot do the job. Postgres held vectors, joins, filters, and transactions in one place; the discipline was refusing to fragment that until we had earned the right to.
And that, looking back across nineteen chapters, was the whole game.
We began with a founder’s uncertainty and a Directus MVP thrown together because shipping and reading real reports beat theorizing — about two thousand users, no idea yet whether anyone would pay. We learned they would, one CCAvenue payment at a time. We strangled the MVP onto Django and DRF rather than betting the company on a rewrite. We split one backend across twenty-odd frontends behind a single Keycloak realm, gave payments their own queue so they never waited behind a video transcode, and put booking correctness in the database where concurrency actually lives. We took endpoints from thirty seconds to two hundred milliseconds with nothing more exotic than the right indexes and honest select_related. We built our own streaming, watched a single day of CloudFront egress hit fifteen hundred dollars and remembered we are an NGO, then chased free bandwidth through Wasabi and Cloudflare and finally Bunny. We tried serverless GPUs, gave up on them, and put a six-hundred-dollar RTX 4050 desktop on a shelf to transcode a hundred and fifty videos a day. We moved terabytes live without dropping a stream.
Every one of those chapters is the same move as this one: ship the humblest thing that works, measure it honestly, and add the next system only when the current one has genuinely run out of room. Not the most impressive architecture — the most appropriate one, chosen with the bill and the on-call rotation in full view.
The search box understands Hindi now. The platform serves thirty thousand people who mostly could not have afforded the alternative. And it is still, underneath all of it, one Postgres, a handful of well-drawn boundaries, and a refusal to add complexity before the moment demanded it.
That was always the job.