Contents
Here’s a requirement that sounds simple until you sit with it: every tenant on a whitelabel platform should be able to pick their own LLM vendor and bring their own API keys. One customer runs everything through Claude, the next insists on GPT for compliance reasons, a third wants Gemini because that’s who their cloud contract is with, and someone eventually asks for Perplexity because they want search-grounded answers. Same codebase, same deploy, four different providers billed to four different accounts.
This was the AI layer of a startup-fundraising platform I spent the better part of four years on. It began as one venture firm’s internal tool and got whitelabeled in mid-2024, when other companies wanted to run it as their own branded product. On paper I was the backend engineer; in practice I was deciding what the product should become, and whitelabel is what forced the multi-vendor LLM abstraction into existence. This post is how I built it, the unconventional call at the center of it, and the isolation traps that bite you if you get the caching wrong.
The conventional move, and why I skipped it
The reflex when someone says “abstract over four LLM providers” is to reach for a framework — LangChain, LiteLLM, or an AI gateway proxy that speaks one API and fans out to many. Those are fine tools. I didn’t use one, and the reason is worth stating because it’s the whole design philosophy.
I only needed one verb. The platform’s AI features — summarizing company research, generating deal-analysis report sections — all boiled down to “give this model a prompt, get text back.” I didn’t need agents, memory, tool-calling orchestration, or a router that ranks models by cost. Pulling in a framework to get complete(prompt) -> str means inheriting its abstractions, its version churn, and its opinions about the ten things I wasn’t doing. When the surface area you actually use is one method, a dependency that ships a hundred is a liability, not a shortcut.
So the interface is deliberately tiny:
from typing import Protocol
class LLMBackend(Protocol):
def complete(
self,
prompt: str,
*,
system: str | None = None,
max_tokens: int = 4096,
) -> str:
...
That’s the entire contract every vendor has to satisfy. A Protocol rather than an ABC on purpose — structural typing means each adapter is just a plain class, no base to inherit, nothing to register.
One adapter per vendor
Each provider gets a thin adapter whose only job is to translate my one method into that SDK’s shape. This is where you learn, viscerally, that “they all take a prompt and return text” is a comfortable lie. The message formats differ, system prompts are handled differently, Claude wants max_tokens as a required top-level field while others treat it as optional, and Perplexity’s models return search-grounded answers with citations you have to decide whether to keep. The adapter is exactly where those differences go to die so the rest of the codebase never sees them.
class ClaudeBackend:
def __init__(self, api_key: str, model: str = "claude-sonnet-4-5"):
self._client = anthropic.Anthropic(api_key=api_key)
self._model = model
def complete(self, prompt, *, system=None, max_tokens=4096) -> str:
resp = self._client.messages.create(
model=self._model,
max_tokens=max_tokens,
system=system or anthropic.NOT_GIVEN,
messages=[{"role": "user", "content": prompt}],
)
return resp.content[0].text
class OpenAIBackend:
def __init__(self, api_key: str, model: str = "gpt-4o"):
self._client = openai.OpenAI(api_key=api_key)
self._model = model
def complete(self, prompt, *, system=None, max_tokens=4096) -> str:
messages = []
if system:
messages.append({"role": "system", "content": system})
messages.append({"role": "user", "content": prompt})
resp = self._client.chat.completions.create(
model=self._model,
max_tokens=max_tokens,
messages=messages,
)
return resp.choices[0].message.content
Gemini and Perplexity get the same treatment — a class, an __init__ that takes a key, a complete that speaks their dialect. Perplexity’s OpenAI-compatible endpoint means its adapter is almost a copy of the OpenAI one with a different base URL and a sonar-family model, which is a nice payoff of keeping the interface narrow.
A thing I want to be honest about: this normalization is leaky. Token limits differ, JSON-mode support differs, refusal behavior differs. I resisted building a grand unified complete that papers over all of it. The interface normalizes the 90% that genuinely is the same; the 10% that isn’t stays visible in the adapter, instead of hidden behind a fake abstraction that lies about what the models can do.
Config lives on the tenant
The vendor choice and the keys aren’t environment variables — they’re columns on the tenant. Each whitelabel customer is a SiteCompany, and two fields carry the entire AI configuration:
class SiteCompany(SiteModel):
llm_vendor = models.CharField(max_length=20, default="claude")
keys = models.JSONField(default=dict) # {"claude": "...", "openai": "..."}
default="claude" is a small opinion that reflects the product: Claude was the strongest default for the summarization and report work, so a tenant who never touches the setting gets a good experience for free. The keys JSONField holds whatever credentials that tenant supplied, namespaced by vendor. Switching from Claude to GPT is one CharField change — no deploy, no per-customer branch.
That bring-your-own-keys decision is the genuinely product-y one, and it wasn’t obvious. The easy path is to run everyone’s AI through my account and bill it back as usage. I didn’t, because a whitelabel customer running deal analysis on their own founders’ pitch decks does not want that data flowing through my vendor relationship, my billing, or my compliance posture. They want their own account, their own invoice, their own data-processing agreement with the provider. Making keys a per-tenant input instead of a platform secret was the difference between a product a regulated customer could adopt and one they’d bounce off in the first security review.
Resolving a client — and the isolation trap
A factory turns a tenant into a ready-to-use backend:
_BACKENDS = {
"claude": ClaudeBackend,
"gpt": OpenAIBackend,
"gemini": GeminiBackend,
"perplexity": PerplexityBackend,
}
def client_for(tenant: SiteCompany) -> LLMBackend:
vendor = tenant.llm_vendor
key = tenant.keys.get(vendor)
if not key:
raise MissingLLMKey(f"tenant {tenant.pk} has no key for {vendor}")
return _BACKENDS[vendor](api_key=key)
Adding a fifth vendor is one adapter class and one dictionary entry. That registry is why, when a customer asked for a provider we didn’t yet support, it was an afternoon’s work rather than a refactor — a quick turnaround that only exists because the interface stayed small.
Now the trap. The tempting optimization is to cache clients so you’re not rebuilding an SDK object on every call. If you cache them keyed on vendor — @lru_cache on "claude" — you have just built a cross-tenant data leak. Tenant A’s request warms the cache with A’s key; tenant B, also on Claude, gets handed A’s client and bills A’s account for B’s work. In a multi-tenant system the cache key must include the tenant, or you don’t cache at all.
The other half of isolation is never falling back to a platform key. Notice client_for raises when a tenant has no key. The seductive alternative — “if the tenant hasn’t configured one, use ours” — silently routes a customer’s private data through your account and puts it on your bill. A missing key is a configuration error the tenant needs to fix, loudly, not a hole to paper over. I’d rather a feature return a clear “configure your AI provider” message than quietly do the wrong, expensive, data-leaking thing.
Tenant resolution itself rides on middleware that stashes the current SiteCompany for the request, so callers deep in a Celery task don’t thread a tenant argument through five layers — they ask for the current tenant and get a correctly-keyed client. That ambient convenience is exactly why the caching discipline above matters: a client cached on the wrong key leaks without anyone passing a wrong argument anywhere.
One design note: keys in a JSONField is convenient but it’s plaintext credentials at rest in your database. Layer encryption on that column — a Fernet-backed field or your cloud KMS — before it holds real customer keys. Convenient storage and secure storage are not the same decision.
What the four-vendor abstraction actually taught me
The lesson that stuck isn’t about LLMs — it’s about where to put a seam. The right abstraction here was smaller than the frameworks on offer, not bigger. One method, one adapter per vendor, config on the tenant, a factory that refuses to guess. Everything hard about four providers collapsed into four small classes that each do one translation, and everything hard about multi-tenancy collapsed into “the cache key is the tenant, and there is no fallback key.”
Whitelabel forced the abstraction, but bring-your-own-keys is the decision I’d point to as product intuition doing its job. It cost more up front — key management, per-tenant config, encryption, clear errors on a missing key — and it’s why the platform could say yes to customers who’d otherwise have walked. Building the AI layer of a multi-tenant product: resist the framework, keep the interface to the verbs you actually use, and treat tenant isolation as a correctness property you defend on every call — not a feature you bolt on later.