Multi-tenant AI is where a small mistake becomes a data breach

Here is the failure that should keep you up at night if you're building AI features for more than one customer. Tenant A asks your assistant a perfectly ordinary question, and the answer contains a sentence from tenant B's confidential document. Not a hack. Not an exotic attack. Just a query that hit a shared retrieval index nobody scoped by tenant, pulled the most relevant chunk regardless of who owned it, and handed it to the model, which dutifully wove it into a fluent reply. That's a breach you disclose, not a bug you file.

Multi-tenant AI is unforgiving in a way that ordinary multi-tenant apps aren't, because LLM features make data flow in ways that route around the access controls you're used to. Your database has row-level security and it works fine, right up until your retrieval layer reads across it, drops the results into a context window, and lets the model speak freely from material it was never supposed to see. The model respects no permission you didn't enforce before the data reached it.

The retrieval layer is the leak

Almost every cross-tenant incident I can imagine starts in the same place: a vector index that holds every tenant's documents together, queried without a hard tenant filter. Similarity search has no concept of ownership. It returns the closest chunks, and if tenant B's chunk is the closest match to tenant A's question, that's what comes back. The fix is conceptually simple and operationally easy to get wrong: every retrieval is filtered by tenant, always, with the tenant identity coming from the authenticated session and never from anything the user or the model can influence.

You've got two structural choices for how strict to be. A shared index with a mandatory tenant filter is cheaper and simpler to operate, but it has a brutal property: a single missing filter, one code path that forgot the WHERE tenant_id =, is a full cross-tenant leak. The blast radius of one mistake is everything. The alternative is an index per tenant, which costs more in infrastructure and operational overhead but makes the leak structurally hard, because tenant A's queries physically cannot reach tenant B's index. For anything sensitive, I lean toward isolation by construction over isolation by remembering to filter, because "remembering to filter" is a thing humans stop doing reliably under deadline.

Everything downstream of retrieval is a leak surface too

The index isn't the only place tenants bleed together. The prompt cache is one people miss: if you cache a prefix that contains one tenant's data and that cache key is shared across tenants, you've built a leak into your cost optimization. Cache keys have to be scoped per tenant whenever the cached content is tenant-specific.

Permissions inside a single tenant matter as well. Even within one customer, not every user should see everything, and the model will happily surface any document your retrieval hands it. This is the same lesson Microsoft 365 Copilot teaches at enterprise scale: the assistant only enforces the access rules you enforce in retrieval, so your per-user permissions have to be applied at the retrieval step, not assumed.

There's even a cross-tenant angle to prompt injection. In a shared system, a document uploaded by tenant B could carry an injected instruction that, if it ever ends up in tenant A's context, manipulates tenant A's session. Treating retrieved content as untrusted isn't only about external web pages; in multi-tenant systems, one tenant's data is untrusted input relative to another's.

Cost and noise need walls too

Isolation isn't only about data. One tenant's runaway agent loop shouldn't be able to spend another tenant's budget or saturate your rate limits and starve everyone else. Per-tenant quotas, rate limits, and cost attribution belong in the gateway all your model calls pass through, so a single bad actor or a single buggy integration is contained to its own tenant's allowance rather than becoming everyone's outage.

Make tenant a first-class concept, and test it

The discipline that holds all of this together is to treat the tenant identifier as a first-class filter on every retrieval, every cache key, every log line, and every quota, applied by default and derived from the authenticated context rather than passed around as a hopeful parameter. Default to deny: a query with no tenant scope should return nothing, not everything.

And then test it directly, because this is exactly the kind of failure that never shows up in normal usage and shows up catastrophically in an audit. Add a cross-tenant isolation check to your eval suite: seed two tenants with distinctive data, run tenant A's queries, and assert that nothing of tenant B's ever appears. It's a cheap test that catches the one bug you most need to catch. Most of what an LLM app gets wrong is a quality issue you can apologize for later; this one ends up in a breach notification instead.