Subtitle: The moment two AI workloads share an identity, each can read the other’s secrets.

---

The LLM Gateway as a Security Control Plane

Most teams adopt an LLM the same way: an application, an API key, a call to a provider. It works, so they do it again — another app, another key, another direct call. By the fifth integration, the security posture has quietly collapsed. Provider keys are scattered across services and CI systems. Every app egresses to the public internet on its own. There is no single place to see spend, cap a runaway workload, or answer “who called what, with which credential.” Nobody decided this; it accreted.

The fix is architectural, and it’s the first thing I build on an AI platform: a gateway. One service — I use the open-source LiteLLM — sits in front of every model provider and becomes two things at once: the only service that holds provider API keys, and the only service permitted to reach provider APIs. Every consumer — a chat UI, an agentic frontend, an MCP server, a batch job — receives a scoped virtual key minted by the gateway, never a provider key and never the gateway’s own master key.

That single move converts a sprawl of ungoverned integrations into a policy enforcement point, and it’s worth being precise about what “enforcement point” actually buys you.

Credential containment. Provider keys exist in exactly one place. Rotating a provider key is a one-service operation, not a fleet-wide hunt. A compromised application never held a provider key to begin with — it held a virtual key, scoped to specific models and a budget, revocable in isolation without touching anyone else.

Egress control. In this design the gateway is the only workload with broad outbound access to the internet (to reach providers whose IP ranges aren’t stable). Everything else runs in private subnets with no route to a provider at all. That means the set of places an attacker could exfiltrate to is one, well-known, and logged — instead of “every app that ever called an LLM.” Egress is where data leaves; collapsing it to one guarded door is one of the highest- leverage controls in the whole platform.

Attribution and budgets. Because all traffic flows through one service, you get unified spend and usage — per key, per team, per end user — and you can set hard budgets and rate limits at the choke point. A budget on a virtual key is a financial circuit breaker: if an agent loops, or a key leaks, the damage is capped at that key’s budget, enforced by the layer that actually spends the money, not by hopeful logic in the frontend.

A uniform audit surface. One service to log, one place that answers “what happened.” That’s the difference between an investigation and a shrug.

The trade-off — because every control has one — is that the gateway becomes critical infrastructure. If it’s down, everything is down; if it’s compromised, it holds the keys. So it earns the strongest treatment: private placement with no public endpoint, a dedicated least-privilege identity, its master key and salt key held in a secrets manager and never in state or on disk, and its admin UI reachable only through an IP-locked path. You are deliberately concentrating risk in order to govern it — which is a fair trade only if you then guard the thing you concentrated it into. Concentration without hardening is just a single point of failure with extra steps.

There’s also a cost to the pattern that’s easy to miss: latency and a dependency. Every call now takes an extra hop. In practice a good gateway adds single-digit milliseconds and pays for itself the first time you fail over between providers, or swap a model without touching a single application. But you should adopt it knowing it’s a dependency you’re taking on, not a free lunch.

The mental split I keep is this: the frontend answers “who are you”; the gateway answers “what may you spend, on which models, and is it allowed.” Identity lives with the app; policy lives with the gateway. Once that boundary is clear, securing an AI platform stops being a scramble across N integrations and becomes the tractable problem of guarding one well-understood service — and giving everything else a narrow, revocable, budgeted key to talk to it.

That’s not an AI trick. It’s the same discipline we’ve applied to API gateways and service meshes for years, aimed at model calls. The novelty in “AI security” is mostly in the payloads; the architecture that contains it is old, and it works.