botminds.aiRequest a demo
Guide · Private AI deployment

On-premise LLM deployment: from GPU memory to a governed endpoint.

Running a language model on your own hardware is a stack of decisions — memory, serving software, model, licence and operations — each with a cost.

An on-premise LLM keeps prompts, documents and answers inside your own data centre. This guide sizes the hardware from first principles, compares the serving software teams actually use, shows how to choose an open-weight model, and sets out where a cloud API still wins.

Updated · 11 sources

Read the guideHow Botminds does it
The on-premise LLM stack
01
Applications and agentsthe assistants, workflows and tools that send prompts
your users
02
Gatewayone front door: sign-in, routing between models, logging, spend
authaudit
03
Inference serverbatches requests, manages the KV cache, speaks an OpenAI-compatible API
vLLMNIMOllamallama.cpp
04
Model weightsan open-weight model, often quantised to fit the memory you have
BF16FP84-bit
05
Cluster runtimecontainers on Kubernetes, GPU drivers and scheduling
KubernetesCUDAROCm
06
HardwareGPUs with high-bandwidth memory, fast links between them
HBMNVLinkInfiniBand

What is an on-premise LLM?

An on-premise LLM is a large language model whose weights run on hardware you own or control — in your data centre or a private facility — so prompts and outputs never travel to an outside AI provider. Deploying one means sizing GPU memory for the model and its working cache, choosing an inference server such as vLLM, NVIDIA NIM or Ollama, picking an open-weight model whose licence fits, and running it behind a gateway that controls who may use it.

On-premise LLM hardware

Sizing on-premise LLM hardware starts with memory.

A model must fit in GPU memory twice over: once for its weights, and again for the key-value cache that grows with every token of context and every concurrent user. Weights are easy to estimate — parameters times bytes per parameter. The cache is what decides how many people the box can serve.

7–9B

Small

About 16 gigabytes of weights at 16-bit, 4–5 at 4-bit. One data-centre GPU with room for a large cache, or a workstation card.

  • Classification, extraction
  • Fast, cheap to serve
20–35B

Medium

About 40–70 gigabytes at 16-bit. One 80-gigabyte GPU at the smaller end; at the larger end two GPUs, or one with 4-bit weights.

  • General assistants
  • Good quality per watt
~70B

Large

About 140 gigabytes at 16-bit — more than one 80-gigabyte GPU. Two to four GPUs with tensor parallelism, leaving room for the cache — or one GPU with 4-bit weights.

  • Strong reasoning
  • Needs a multi-GPU node
400B+

Very large

Hundreds of gigabytes even when quantised. A full eight-GPU node or several nodes; mixture-of-experts models activate only part of it per token.

  • Frontier-class quality
  • Cluster-scale cost
141
gigabytes of HBM3e memory on one NVIDIA H200, at 4.8 terabytes per second — nearly double the H100’s capacity, by NVIDIA’s own figures
NVIDIA
1 GPU
gpt-oss-120b (117B parameters, 5.1B active per token) fits a single 80-gigabyte GPU such as an H100, because its expert weights ship in 4-bit MXFP4
OpenAI model card
2–4×
throughput gain from vLLM’s PagedAttention at the same latency, against FasterTransformer and Orca, in the authors’ benchmarks
Kwon et al., SOSP 2023

How to read the sizes. The gigabyte figures are our arithmetic for weights alone: parameters times two bytes at 16-bit, one at 8-bit, half a byte at 4-bit. Add the key-value cache and runtime overhead on top, and confirm against the cache size the server reports when it starts.

Inference servers

On-premise LLM serving stacks: vLLM, NVIDIA NIM, Ollama and llama.cpp.

The inference server turns weights into an endpoint. All four below expose an OpenAI-compatible API, so applications written for a cloud model can usually point at your own server with a URL change. They differ in how many people they are built to serve.

One person, one machineMany users, GPU clusters
1234
  1. llama.cppC/C++ inference on CPUs, laptops and single GPUs; GGUF files quantised from 8 down to 1.5 bits.
  2. OllamaA local server on port 11434 that pulls and runs quantised GGUF models; an OpenAI-compatible subset under /v1.
  3. vLLMOpen-source serving with PagedAttention and continuous batching; tensor parallelism within a node, pipeline parallelism across nodes.
  4. NVIDIA NIMPrebuilt, tuned containers with an OpenAI-compatible API backed by vLLM, plus health and metrics endpoints; production use needs an NVIDIA AI Enterprise licence.
Serving an open-weight model with vLLM
# a ~70B model at 16-bit: one node, four GPUs
vllm serve meta-llama/Llama-3.1-70B-Instruct \
  --tensor-parallel-size 4

# then, from any OpenAI client
POST /v1/chat/completions
{ "model": "meta-llama/Llama-3.1-70B-Instruct",
  "messages": [{ "role": "user",
    "content": "Summarise this contract" }] }

Splitting a model across GPUs

  • Fits one GPURun it there; no distribution needed.
  • Fits one nodeTensor parallelism: each layer split across that node’s GPUs.
  • Needs several nodesTensor parallelism inside each node, pipeline parallelism across nodes.
  • GPUs without NVLinkPipeline parallelism, which talks less between cards.
Open-weight models

Choosing an open-weight model: size, licence and fit.

Open-weight models now cover most enterprise tasks, from extraction to multi-step reasoning. Two questions narrow the field fast: will it fit the hardware you have, and does its licence allow your use?

Fits one GPU
gpt-oss-20b and 120bApache 2.0. The 20b runs within 16 gigabytes; the 120b fits one 80-gigabyte GPU.start here
Llama 3.1 8B and 70BMeta’s community licence, with an acceptable-use policy and a clause for services above 700 million monthly users. The 8B fits any data-centre GPU; the 70B fits one 80-gigabyte GPU once quantised to 4-bit.read the terms
Needs a multi-GPU node
DeepSeek-R1MIT licence; a 671B mixture-of-experts model that activates 37B per token — a full node or more.cluster
Llama 3.1 405BThe same community licence at frontier scale; hundreds of gigabytes even quantised.cluster
Permissive licenceCustom licence terms
Before you commit to a model
  • Test on your own tasksScore it on a golden dataset of your documents and questions, not a public leaderboard.
  • Check the licenceCommercial use, user thresholds, acceptable-use terms and whether outputs may train other models.
  • Check the formatPrefer safetensors or GGUF from the publisher; formats that can run code on load are a supply-chain risk.
  • Measure the cacheContext length times concurrent users decides memory as much as model size does.
  • Plan the next modelA gateway in front lets you swap models without touching the applications.
On-premise LLM vs cloud

On-premise LLM vs cloud API: what you gain and what you take on.

Cloud model APIs are quick to start, scale without hardware and get new models first. An on-premise LLM trades that for control. The decision is usually made by data rules first and cost second.

your hardware

On-premise LLM

Weights run in your facility. Prompts and outputs stay inside your network.

  • Data never leaves your site
  • Works with no internet link
  • Hardware bought up front; power and staff after
  • You run, patch and scale it
a provider’s hardware

Cloud model API

You call a hosted model over the network and pay per token or reserved capacity.

  • Newest and largest models first
  • No hardware to buy or run
  • Cost follows usage
  • Data processed under the provider’s terms and regions
Your data centreWhere prompts are processedThe provider’s regions
Open-weight modelsModel choiceThe provider’s catalogue, including closed models
Capital up front; cheap when busyCost shapePer token; cheap when idle
Buy, rack and power GPUsScalingRequest quota
YesDisconnected sitesNo
Your teamOperationsThe provider

Which deployment fits your workload?

Data may not leave the site, or the site has no internetAn on-premise LLM is the only option. Size for peak concurrency and plan the update path across the airlock.
Data must stay in a country or region, but a cloud is acceptableA cloud model deployed in your own account and region, reached privately, usually meets the rule with far less to run.
Steady, heavy volume on a known taskRun the numbers: busy GPUs on your floor can undercut per-token prices. Keep a cloud model for peaks and new capabilities.
None of theseStart with a cloud API behind a gateway, so moving a workload on-premise later is a configuration change.
Step by step

Rolling out an on-premise LLM deployment in five steps.

A single model server can be answering requests within a day. Making it something hundreds of people rely on takes the rest of the stack: a gateway, monitoring and a way to change models safely.

  1. Pick the task

    Name the workload, its volume, its latency target and the data rules it must follow.

  2. Shortlist models

    Two or three open-weight candidates that fit the licence; score them on your own golden dataset.

  3. Size the hardware

    Weights plus cache at peak concurrency, with headroom; decide GPUs per node and nodes.

  4. Serve it behind a gateway

    Run the inference server in containers on Kubernetes with health checks and metrics, and put a gateway in front for sign-in, per-team access, routing, logging and spend.

  5. Operate

    Watch latency, cache use and errors; re-score on every model or prompt change; patch on a schedule.

On Botminds

How Botminds runs agents on an on-premise LLM.

On Botminds, models are records you configure, and every call goes through the model gateway inside your deployment. Point the gateway at a model on your own hardware and agents use it like any other model.

Botminds on your premisesthe same release as the cloud, on Kubernetes in your data centre
Your own modela deployment on your hardware, reached through the gateway
Or your cloud accountAzure OpenAI or Azure AI Foundry in your subscription and region, Bedrock in your AWS account, or Google Gemini
One gatewaythe one service that holds the provider key
Disconnected sitesno inbound access, no VPN to us; AI stays on site
Signed releaseschecksummed bundles of verified images, carried across your airlock
Private endpointsin a dedicated Azure environment, data services and the model account sit on private endpoints
Your regiondocuments, indexes and records stored where you choose
No training on your datayour documents never train shared models

Read the deployment shapes in private AI deployment, and how models are scoped and routed in the LLM gateway.

Questions people ask

On-premise LLM deployment: common questions.

The technology behind itPrivate AI deploymentAzure, AWS, Google Cloud or on-premises
What hardware do I need for an on-premise LLM?

For an on-premise LLM, enough GPU memory for the model’s weights plus its key-value cache. A rough guide for weights: two bytes per parameter at 16-bit, half a byte at 4-bit — so a 70B model needs about 140 gigabytes at 16-bit, or roughly 35–40 when quantised to 4-bit.

Can you run an LLM locally without a GPU?

Small quantised models run on CPUs with llama.cpp or Ollama, which is fine for one person or a pilot. Serving many users at useful speed needs data-centre GPUs and a batching server such as vLLM.

What is the best software to serve an LLM on-premise?

For many users, vLLM or NVIDIA NIM — both batch requests efficiently and expose an OpenAI-compatible API. For a single machine, Ollama or llama.cpp. Pick by concurrency, hardware and the support you need.

Is an on-premise LLM cheaper than a cloud API?

It can be when GPUs stay busy on steady, heavy volume, because the hardware is paid for once. For spiky or light use, per-token cloud pricing usually wins. Include power, cooling, staff and refresh cycles in the comparison.

Which open-weight models can run on-premise?

Many: OpenAI’s gpt-oss models (Apache 2.0), Meta’s Llama family (community licence), DeepSeek-R1 (MIT), and others. Check each licence’s terms and test the model on your own tasks before choosing.

What is the difference between on-premise and private cloud LLM deployment?

On-premise means hardware you own in a facility you control. A private cloud deployment runs a model in your own cloud account and region. Both keep data under your rules; only on-premise works with no outside connection.

Does vLLM support an OpenAI-compatible API?

Yes. The vLLM server exposes chat completions, completions, embeddings and model listing endpoints in the OpenAI format, so most OpenAI clients work by changing the base URL.

Can Botminds use an on-premise LLM?

Yes. Point the Botminds model gateway at a model deployment on your own hardware, and agents use it like any other model. The platform itself runs on Kubernetes in your data centre, including disconnected sites.

Sources
  1. vLLM documentation: OpenAI-compatible serverdocs.vllm.ai
  2. vLLM documentation: parallelism and scalingdocs.vllm.ai
  3. Kwon et al.: Efficient Memory Management for LLM Serving with PagedAttention (SOSP 2023)arxiv.org
  4. NVIDIA NIM for LLMs: API referencedocs.nvidia.com
  5. NVIDIA: Run NIM anywheredocs.api.nvidia.com
  6. Ollama: OpenAI compatibilitydocs.ollama.com
  7. llama.cppgithub.com
  8. NVIDIA H200 Tensor Core GPUnvidia.com
  9. OpenAI: gpt-oss-120b model cardhuggingface.co
  10. DeepSeek-R1 on Hugging Facehuggingface.co
  11. Meta: Llama 3.1 70B Instruct and licencehuggingface.co
Read next
Sovereign AIWho controls the data, models, compute and law under an AI systemData residency for AIResidency, sovereignty and localization, GDPR transfers and model callsAI agent security best practicesIdentity, least privilege, approvals, containment and audit for agentsAll guidesEvery explainer, by topic

See it on
your own work.

Bring one process and its documents. We’ll show you the agents, the controls and the record, running on your data.

Book a demoPrivate AI deployment