your deployments
Azure OpenAI
Your Azure OpenAI deployments by endpoint, deployment name and API version — reasoning models included.
LLM gateway
The LLM gateway: every model behind one door.
Azure, Google, OpenAI, AWS Bedrock or models you host — routed, governed and costed in one place.
Every agent, workflow step and mesh on Botminds reaches its language model through one gateway. It finds the model, checks the allowance of the person’s role before the run starts, spreads calls across providers and regions, holds the keys, moves past a model in trouble and records what every run cost. Changing a model is a setting, not a rewrite.
Callers
- Agents
- Workflow steps
- Meshes
- Piper
- Your systems
One gateway to every model
- Resolvethe model or pool: project, then workspace, then environment
- Admitthe allowance of the person’s role, checked before the run starts
- Spreadacross deployments, providers and regions, at each one’s pace
- Callthrough the gateway — the one holder of the key
- Recordtokens and cost, by team, workspace and project
Models
- Azure OpenAI
- Azure AI Foundry
- Google Gemini
- Any model on the gateway’s list — OpenAI, AWS Bedrock or models you host
What is an LLM gateway?
An LLM gateway is one layer between your applications and every language model they call. It routes each request to an approved model, holds the provider keys, paces and spreads traffic, fails over when a provider has trouble and records what each call cost. The Botminds LLM gateway does this for every agent on the platform — Azure OpenAI and Azure AI Foundry deployments, Google Gemini, and any model added to the gateway’s model list — OpenAI, AWS Bedrock or models you host — so they are all used, scaled and governed the same way.
Scales out on KubernetesServices and workers add capacity as work arrives, and give it back when it is done.
Built for millionsMillions of documents, records and agent runs — the indexes and workers grow with them.
Any cloud, any regionAzure, AWS, Google Cloud or on-premises, in the region your data must stay in.
Always onRedundant services, rolling releases and continuous backups, for mission-critical work.
Secure and governedSingle sign-on, encryption, least privilege and a sealed audit trail.
Why it matters
Why put an LLM gateway between agents and models?
Fifty agents across ten teams, four providers and a finance department asking what it all costs: without one layer in the middle, every application carries its own keys, its own retries and its own idea of a quota.
| Each app calls models directly | Botminds LLM gateway | |
|---|---|---|
| Provider keys | Copied into every app and every script | Held by the gateway, encrypted |
| Changing a model | A code change in each app | A setting on the agent, or the default for a whole scope |
| A provider or region has trouble | Each app fails, or retries on its own | On document and background runs, the pool moves the call to another provider or region |
| Traffic bursts | Every app bursts, and the provider refuses | Paced to each deployment’s declared quota and spread across the pool |
| Spend | One provider invoice, no breakdown | Cost by team: per run, agent, project and workspace |
| Who may use what | Whoever has the key | Models scoped to a project, a workspace or the environment |
Bring your own model
Every model your enterprise uses, through one gateway.
Multi-model AI is the normal case: a fast model for conversation, a reasoning model for analysis, an embedding model for search — often from different providers. Add each deployment to the gateway once, and every agent can use it.
your deployments
Azure AI Foundry
Models deployed in Azure AI Foundry, governed by the same controls as everything else.
Gemini
Gemini models connect with their own key and a thinking budget, or join the gateway’s model list.
on the model list
OpenAI
OpenAI’s own models on your account and quota, added to the gateway’s model list.
on the model list
AWS Bedrock
Models served by AWS Bedrock in the AWS regions you use, added to the gateway’s model list.
your site
Models you host
Your own model deployment, behind your network, added to the gateway’s model list. Agents use it like any other model.
A model is a record, not code. Provider, deployment, quota, chat or embedding, and whether it is the default for its scope. Agents point at the record, so moving every agent in a project to a newer model is one edit — and moving back is another.
Who may use which model
Model scopes: per project, workspace and environment.
A bank’s retail team and its research team rarely share one model account. Each team’s deployments — a region its regulator requires, say — are scoped to its workspace or project, while the environment keeps a catalogue everyone can use.
Environmentset by platform administrators
the catalogue every workspace may use, and its default chat model
Workspaceset by a business unit’s administrators
its own models — its own Azure deployment, say, or a model added to the gateway’s model list
Projectset by the project’s builders
models one program needs, such as a deployment in a required region
Agentnames a model, or a pool of several — or none, and gets the default
How an agent’s model is found
- Projectlooked up first
- Workspaceif the project has no such model
- Environmentif neither has it
The first match wins. Each scope can mark one model as its default, so an agent that names none still runs.
Models travel with solutions. A model record travels in a solution package with its secrets stripped at export, and is mapped to a model in the target environment on import.
Model fallback
Model pools: fallback across providers and regions.
Name several models on an agent — in different regions, from different providers — and they become a pool. Every call draws from it; on document and background runs, a model in trouble is set aside and the call moves on. The agent and its instructions stay exactly as they are.
One agent’s pool
- Azure OpenAIWest Europe
- Azure OpenAISweden Central
- Azure AI FoundryNorth Europe
- A model you hostyour network
Each attempt draws at random from the models in play, so load spreads across providers, regions and quotas.
- RequestWest EuropeThe region is having trouble and the provider returns a service error.set aside
- Same requestSweden CentralDrawn from the models still in play — another region, another quota.answered
- Next requestWhole poolStarts with every model back in play, so a region that recovers takes traffic again.spread
What moves to another model
- Provider and service errorsThe next attempt draws from the models that have not failed in this request.
- A refusal on a model with declared limitsCalls are paced to the declared limits; a refusal moves to a different deployment in the pool — a different quota.
What waits, and what reports at once
- A refusal on a model without declared limitsThe call waits for the provider’s reset time, then goes again.
- A refusal with nowhere to moveReported with its cause, so you see which quota to raise or which model to add.
- A request the model rejects as it standsReported at once with its cause, in plain words, so the fix is obvious.
Rate limits
Rate limits: bursts are paced and spread, not refused.
The usual answer to a rate-limit refusal is to retry at once. When every layer does, one refusal becomes a storm. The platform does the opposite: declare each deployment’s quota and it sends work at the pace the models can take.
Declared once, on the model
- Tokens a minuteas the provider’s deployment declares
- Requests a minutethe same
- Calls in parallelwhat it serves at once
sizeAdmit
How much document work runs at once is sized from the quota and the measured cost of a document. Everything else waits in a durable queue.
pacePace
Every process sends calls at its share of the quota. A burst flows out as a steady stream instead of drawing a storm of refusals.
moveRecover
A refusal moves the call to another deployment in the pool, or is reported with its cause. On a model without declared limits, the call waits for the provider’s reset and goes again.
One owner for every kind of fault. Model client libraries retry by default; stacked under a gateway and an agent loop, polite layers multiply one fault into dozens of calls. On Botminds each kind of fault has one owner, and every retry that fires is counted — so a pattern shows up as the defect it is.
At scale
An LLM gateway built for internet-scale traffic.
Thousands of agents and millions of documents mean a torrent of model calls. The gateway is stateless, so it scales out horizontally on Kubernetes; it spreads load across every deployment and region you give it; and it keeps serving when a provider or a region has trouble.
Callers · scale out on their own queues
- Agents
- Workflow steps
- Mesh workers
- Your systems
The gateway · one addressstateless copies on Kubernetes, one shared catalogue and spend ledger
- copy
- copy
- copy
- copy
- + more as traffic grows
Model capacity · providers and regions
Region Aserving
- Azure OpenAI
- Azure AI Foundry
Region Brerouted
- Azure OpenAI
Model listserving
- OpenAI
- AWS Bedrock
- Google Gemini
Your siteserving
- Models you host
Scales out horizontallyMore gateway copies start as traffic grows, and are released as it falls.
Spreads the loadCalls fan out across deployments, providers and regions — every quota working at once.
Keeps servingWhen a provider or a region has trouble, traffic moves to the rest of the pool.
Model settings
Sampling, reasoning effort and thinking budgets, per agent.
A classifier must be repeatable; an assistant drafting emails should not be. Each agent carries its own model settings, and the platform adapts them to the model family before the call.
{
"models": ["fast-eu-west", "fast-eu-north"],
"temperature": 0,
"top_p": 1,
"seed": 42
}Prompt caching. Agents on gateway models send a stable cache key, so providers that cache serve the repeated part of long instructions at a lower price.
- SamplingTemperature, top-p, frequency and presence penalties, and a seed for repeatable runs.
- Reasoning effortFor reasoning models: how hard to think, sent the way those models expect.
- Thinking budgetFor models that take one: how much thinking to do before answering.
- Adapted to the modelEach setting is translated for the model family, so one agent definition runs on any model in its pool.
LLM cost control
LLM cost control: allowances before the run, cost by team after it.
A budget enforced after the spend is a report. The platform checks the allowance at the one door every run passes — chat, workflow step, schedule or API call — and attributes every token to the team that spent it.
Before
Admit
Checked against the allowance of the person’s role before a token is spent; a monthly allowance per person can be set for the assistant.
- Chat, workflow steps, schedules and API calls pass the same check
- An over-allowance run is told why, in plain words
During
Tag
Every call carries its workspace, its project and the kind of traffic, so no spend arrives unattributed.
- Platform overhead tagged apart from your agents
- A missing tag shows up as a visible gap
After
Price
Tokens in, cached and out are priced from one table and written to two ledgers that must agree.
- Run record: cost per run, agent and project
- Gateway ledger: spend by team — workspace and project
Cost by team, from one price table. Every call carries its workspace and project, and is priced once. Finance sees spend by business unit and program; builders see cost per run and per agent. Access itself is decided by roles and grants — see AI governance.
Private LLM
Private LLM: your keys, your deployment, your region.
For many enterprises the question is not which model, but where it runs and who can see the traffic. The gateway runs inside your Botminds deployment, and the models behind it can be yours.
Your own deployment
A model you run — in your region, behind your network — joins the gateway, and agents use it like any other.
- Azure OpenAI or Foundry in your subscription
- Bedrock models on the gateway’s model list
- Models you host on your own hardware
One holder of the key
In production the gateway holds the provider key. Agents, workers and the API reach every model through it, and a workspace can carry a gateway key of its own.
Content stays out of billing
The spend ledger keeps tokens, cost and tags. Prompts and answers stay in the run record, inside your deployment, under your access rules.
Run the whole platform privately. Azure, AWS, Google Cloud or your own premises, down to fully disconnected sites: see private AI deployment. Your documents never train shared or third-party models — see security.
Side by side
LLM gateway vs AI gateway vs LLM router: what each decides.
An LLM router picks a model per prompt; a standalone AI gateway is a proxy you deploy and re-point every application at; an LLM gateway inside the platform knows which agent, project and person each call belongs to.
| LLM router | Standalone AI gateway | Botminds LLM gateway | |
|---|---|---|---|
| What it decides | Which model answers this prompt | Where a request goes, and whether it may | Which model each agent uses, and how every call is made |
| Knows who is asking | No | An API key | The agent, the project, the workspace and the person |
| Provider keys | Yours to manage | Held by the gateway | Held by the gateway, encrypted |
| Fallback | Sometimes, by model score | Configured per route | A pool per agent, across providers and regions |
| Rate limits | Its users’ problem | Refuses over the limit | Paces to each quota and spreads across the pool |
| Spend | Its users’ problem | Per key | Allowance before the run; cost by team, per run and agent |
| Choosing a model | Per prompt, by predicted score | Per route | Per agent, proven by evaluations |
| Wiring | In your code | Every app re-pointed | Built in; every agent has it |
Questions people ask
LLM gateway, answered.
More from the platform: the full architecture, and how security works at Botminds.
What is an LLM gateway?
An LLM gateway is a single layer between applications and the language models they call. It routes requests to approved models, holds the keys, paces and spreads traffic, fails over when a provider has trouble and records cost. In Botminds the LLM gateway is built into the platform, so every agent uses it.
What is the difference between an LLM gateway and an AI gateway?
The terms overlap. “AI gateway” usually means a general proxy that sits in front of AI services and applies routing and policy per API key. The Botminds LLM gateway also knows which agent, project, workspace and person each call is for.
What is the difference between an LLM gateway and an LLM router?
An LLM router chooses which model should answer each prompt, often by predicted quality or price. A gateway governs how calls are made: keys, pacing, fallback and cost. Botminds sets a model or a pool per agent and proves the choice with evaluations.
Which models can I use with Botminds?
Azure OpenAI and Azure AI Foundry deployments, Google Gemini, and any model added to the gateway’s model list — OpenAI, AWS Bedrock or models you host. Add each one once; scopes then decide which project, workspace or environment uses it.
How does model fallback work?
Name two or more models on an agent — in different regions or from different providers — and they form a pool. Each call draws from it; on document and background runs, a model that fails is set aside for the rest of that request and the call moves to another. The next request starts with the whole pool again.
What happens when a model hits its rate limit?
When a model’s limits are declared, calls are paced to them; a refusal moves to another model in the pool, or is reported with its cause. On a model without declared limits, the call waits for the provider’s reset and goes again. A burst flows out steadily, and every retry is counted.
How does the LLM gateway scale?
The gateway is stateless, so it scales out horizontally on Kubernetes as traffic grows, with every copy sharing one model catalogue and one spend ledger. It spreads calls across deployments, providers and regions, and keeps serving when one of them has trouble.
How do you control LLM costs?
Allowances are checked before a run starts, at the one door every run passes. Every call is tagged with its workspace and project and priced from one table, so spend is visible by team, by program, per agent and per run.
Can different teams use different models?
Yes. Models live at three scopes — environment, workspace and project — and an agent’s model is found in its project first, then its workspace, then the environment. A team’s own deployment is scoped to its workspace or project, without touching anyone else’s.
Can I run a private LLM with Botminds?
Yes — Azure OpenAI in your own subscription and region, or Bedrock models and models you host on your own hardware added to the gateway’s model list. The platform itself can run in your cloud or on your premises. See private AI deployment.
Are my documents used to train the models?
No. Your documents and prompts never train shared or third-party models. See security at Botminds.
Every model, governed in one place.
Bring your model accounts. We’ll show your agents running across them, with pools, pacing and cost by team.