An LLM gateway governs calls to language models. An AI gateway puts the same controls in front of models, MCP tool servers and agent APIs alike.
Both sit between your applications and the AI services they call. This guide places each on one scale of scope, follows a model call through a gateway step by step, lists the jobs a gateway must do on every call, and shows how to decide which layer — or which combination — your systems need.
Updated · 7 sources
An LLM gateway is a proxy for language model calls: it routes each request to an approved model, holds the provider keys, enforces token quotas, fails over between deployments and records what each call cost. An AI gateway applies those controls to all AI traffic — models, MCP tool servers and agent APIs — and is often an extension of an existing API gateway. In LLM gateway vs AI gateway, the difference is scope: every LLM gateway job is part of an AI gateway, but not the reverse.
Vendors use the two names loosely, and some products carry both. The useful distinction is what traffic each one is built to understand.
A single endpoint in front of many language model deployments. It speaks the chat, responses and embeddings APIs and understands tokens.
The control point for every AI service an organisation runs: models, the tools agents call and the agents themselves.
One product, two names. Microsoft describes the AI gateway in Azure API Management as a set of capabilities that extends its existing API gateway, not a separate product, across language model APIs, remote MCP servers, agent-to-agent APIs and self-hosted models. Cloudflare’s AI Gateway documentation covers model traffic only — analytics, logging, caching, rate limits, retries and model fallback. Kong’s AI Gateway covers model traffic and also exposes and governs MCP tools and agent-to-agent traffic. The same name spans both scopes.
Every call an agent makes passes identity and policy first. Model calls then get quota, cache, deployment choice and fallback; tool and agent calls get routing and sign-in on the caller’s behalf.
Strip the product names away and every LLM gateway does the same core jobs. An AI gateway does all of them, then repeats the identity and logging jobs for tools and agents.
Spillover saves money. Microsoft’s architecture guidance describes slightly under-buying reserved (provisioned) capacity and placing a pay-as-you-go deployment beside it, so the gateway sends overflow there. Clients see one endpoint; the bill sees full use of the reserved capacity.
The order matters. A gateway that checks quota after the call has already spent the tokens; one that retries a refusing backend at once turns one fault into many.
The gateway learns who is calling — a key, an application, or a signed-in person.
It counts the prompt’s tokens and refuses up front if the caller’s quota would be exceeded.
A matching earlier answer returns without a model call.
Reserved capacity first, then pay-as-you-go; a nearby region before a far one.
The app never holds the provider key.
A 429 (Too Many Requests) may carry a Retry-After header (RFC 6585). The gateway breaks the circuit to that deployment for that long and moves the call to another one, rather than hitting the refusing endpoint again.
Tokens in, tokens out, cached tokens, model, caller and cost go to one ledger.
Two questions settle it: how many model providers and deployments you run, and whether your agents call tools and other agents that also need governing.
What problem are you solving first?
Most gateways demo the same dashboard. The differences show up under load, under failure and in the audit.
A standalone gateway knows the caller you configure it to see — a key, an app, a signed-in identity. The gateway built into Botminds knows which agent, project, workspace and person every call belongs to, so routing, fallback and cost follow your organisation’s structure without re-pointing any application.
| Standalone AI gateway | Botminds LLM gateway | |
|---|---|---|
| Knows who is asking | A key, an app or a signed-in identity | The agent, the project, the workspace and the person |
| Provider keys | Held by the gateway | Held by the gateway, encrypted |
| Fallback | Configured per route | A pool per agent, across providers and regions |
| Rate limits | Refuses over the limit | Paces to each deployment’s declared quota and spreads across the pool |
| Spend | Per key, or per tag you add to each call | Allowance checked before the run; cost per run, agent, project and workspace |
| Who may use a model | Policy written per route and caller | Models scoped to a project, a workspace or the environment |
| Wiring | Every app re-pointed | Built in; every agent has it |
Models are records — Azure OpenAI and Azure AI Foundry deployments, Google Gemini, and OpenAI, AWS Bedrock or models you host added to the gateway’s model list — so moving a project to a newer model is one edit. Read the full design in the LLM gateway, and how agents use models in the Academy lesson LLMs and services. For tool traffic, see MCP for the enterprise.
In LLM gateway vs AI gateway, the difference is scope. An LLM gateway governs calls to language models: routing, keys, token quotas, fallback, caching and cost. An AI gateway applies the same controls to every AI service — models, MCP tool servers and agent APIs.
It gives applications one endpoint for every language model, holds the provider keys, enforces token quotas per caller, balances load and fails over between deployments, caches repeated prompts, and records tokens and cost for every call.
An API gateway handles any HTTP API: authentication, request rate limits and routing. An LLM gateway understands model traffic — it counts tokens, translates between provider APIs, caches prompts and fails over between models. Several AI gateways are built as extensions of an API gateway.
No. A router decides which model should answer a given prompt, often by predicted quality or price. A gateway decides how every call is made — with which key, under which quota, to which deployment, and with what fallback. A router can run inside a gateway.
An MCP gateway sits in front of Model Context Protocol servers, so the tools agents call get the same treatment as model calls: sign-in, policy, routing and logs. Azure API Management and Kong both cover MCP servers alongside model APIs in their AI gateways.
Not always. Microsoft’s architecture guidance says that with a few clients whose code you control, the routing logic can live in that code. A gateway earns its place once many applications share quotas, teams need cost broken out, or you need failover across deployments and regions.
It can. A semantic cache returns the stored answer of an earlier prompt that is close in meaning, so a question that differs in one detail may get the earlier answer. Use it on routes where near-duplicate questions really share an answer, and tune the similarity threshold.
Yes. Every agent, workflow step and mesh on Botminds reaches its model through the built-in LLM gateway, which holds the keys, pools models across providers and regions, paces calls to each quota and records cost by team. See the LLM gateway.
Bring one process and its documents. We’ll show you the agents, the controls and the record, running on your data.