Running LLMs On-Prem for Regulated Industries: A Field Guide

"On-prem" is a regulatory answer, not a technical preference

Every conversation about on-premise language models starts the same way. A bank, a ministry or a hospital group has seen what a good model does with a loan file, a survey dataset or a discharge summary, and has also seen the line in its own policy that says none of those may leave approved infrastructure. The question is not whether a private model can do the job — the open-weight field settled that — but what it takes to run one where the data is allowed to be, with controls a security team will sign off, at a cost that beats the API it replaces.

This is the guide we wish more teams had read before they bought GPUs. It is drawn from our private LLM deployment practice and from the regulated platforms we already operate under data-residency, maker-checker and audit rules. The short version: read the regulator's text first, pick the model on your own tasks, size the hardware from arithmetic, and treat the controls around the model as the product, because they are what gets it approved.

First: which tier does your regulator actually permit?

"Private LLM" covers three quite different setups, and the right one depends on what your regulator and your risk team require, not on what sounds safest.

Tier one — on-premise. Open-weight models on GPUs in your own data centre or a rented rack in your country, with no path to the internet at all. This is the answer for defence, central banks, hospitals and anyone whose data may not leave a named facility.
Tier two — your own cloud account. The same open models served on GPU instances inside your VPC in an approved region, with your keys, your logs and your network controls.
Tier three — an in-region managed endpoint. Amazon Bedrock, Azure OpenAI or Google Vertex AI, where the model is the provider's but the contract guarantees no training on your data and residency in a region you choose.

Most enterprises we talk to assume they need the first tier and are permitted the second or third. The difference is significant: tier one means buying or renting GPUs, running them at useful utilisation and keeping a team on them; tier three means an API call and a contract. So the first two weeks of any engagement are spent reading the regulator's actual text — the Reserve Bank of India's outsourcing and cloud guidance for Indian banks, SAMA's cybersecurity framework for Saudi financial institutions, the DPDP Act and Saudi PDPL for personal data, sector rules for healthcare, or your own internal policy — and writing down which tier it permits for which class of data.

That last clause matters. The same organisation is often permitted different tiers for different data. Customer credit data may have to stay on named premises while marketing copy and internal documentation may go to an in-region endpoint. A single "we are on-prem" decision applied to everything is usually the most expensive way to be compliant. A data-classification table with a tier against each row is the cheapest, and it is the first artefact we produce.

The rest of this guide is about tier one, because it is the tier where the engineering is hardest and where most of the advice on the internet is written by people who have not had to get a deployment past a bank's security review.

What an on-prem deployment actually looks like

Running a model is the easy part; a single command does it. Running one that stays fast at month-end, that nobody outside the perimeter can reach, and that costs less than the API it replaced is engineering. In practice the deployment has a recognisable shape.

The model is an open-weight checkpoint — Llama, Qwen, Mistral, Gemma, DeepSeek or Nemotron families, or a fine-tuned derivative — downloaded from the publisher, verified by checksum and mirrored inside your network. It is served by vLLM in nearly every production deployment we run: continuous batching, paged attention so long contexts do not waste GPU memory, tensor parallelism to split a large model across cards, and an OpenAI-compatible API. That last feature is not cosmetic. The applications, agents and retrieval pipelines you wrote against the API you are replacing move over with a base-URL change rather than a rewrite. For small teams and edge boxes, Ollama is the lighter option.

Around the model: Docker or Kubernetes for packaging, NGINX with TLS in front, your identity provider for access, an audit store you control for the logs, and — where the network is closed — an installation from a signed offline bundle with no outbound access at all. The applications on top are built in Python; the serving layer exposes the same interface either way, so the application team does not need to know which GPU the answer came from.

Choosing the model when you cannot phone a vendor

With an API you rent the vendor's judgement about which model is good. On-prem you own that decision, and the temptation is to take it from a leaderboard. We do not, for the practical reason that leaderboards measure tasks that are not yours, in languages that may not be yours, at precisions you may not be running.

The method that works is a test set drawn from your real work: a few hundred prompts with expected outputs, in every language you need, scored by an LLM judge and spot-checked by your own people. Three to five candidate models and sizes go through it, each at full precision and quantised. Quantisation is part of the test rather than an afterthought — a model quantised to fit a smaller GPU is a different model, and the quality loss has to be measured, not assumed. Latency and tokens per second are recorded on the target GPU, not on a laptop.

Licences are checked before anything is downloaded. Most open-weight models are commercially usable; some carry attribution or usage restrictions that legal needs to see in writing. The output of the exercise is a decision record — model, precision, hardware, scores and cost per request — that is kept for the auditor, because in a regulated deployment "why this model" is a question you will be asked, possibly years later, by someone who was not in the room. The Hugging Face hub is where the candidates come from; the decision record is what turns a download into something defensible.

Sizing the GPUs: arithmetic, not a catalogue

The single most expensive mistake in on-prem LLM projects is buying hardware from a catalogue before anyone has done the arithmetic. The arithmetic is not complicated. Your expected concurrent users, the length of a typical prompt and answer, and the latency your users will tolerate give a required token throughput. The model, its precision, and the GPU's memory and bandwidth give a delivered throughput per card. The ratio, with headroom for peaks, is the number of GPUs.

As a rule of thumb from our deployments: a quantised 7–9-billion-parameter model serves a department from a single 24 GB card; a 70-billion-parameter model wants two 80 GB cards or four 48 GB ones; anything larger is a cluster and a conversation about whether tier three is the better answer for that workload. The sizes we serve run from 8B to 70B and beyond, quantised to fit the GPUs you have rather than the GPUs the model would prefer.

Two things the catalogue does not tell you. First, utilisation: a GPU that is busy for two hours a day is an expensive way to run a model, and the cost model has to reflect the traffic you will actually have, not the peak you are afraid of. Second, procurement: in India and the Gulf, buying GPUs can take longer than building the deployment, which is why we also deploy on GPU capacity rented from in-country data-centre providers where the regulator permits a rented rack. Where the network must be closed, the rented rack is still an option; the constraint is the network path, not the ownership of the metal.

The controls the security review will ask about

A private model earns its keep only if the controls around it are as strong as the isolation. This is the part of the project that gets a deployment approved, and it is the part most teams under-scope. The controls we build and document, in the order a security reviewer tends to ask about them:

Network isolation. The inference hosts sit in a segmented network with no outbound access. Nothing on the GPU box can call home, fetch a package or phone a telemetry endpoint.
Identity and roles. Access is through your identity provider, with roles for who may call which model with which class of data. A model that can see credit files is not the same endpoint as the one that drafts internal emails.
Per-request audit logging. Every request and response is logged with user, timestamp and model version to an audit store you control, with retention you set. If a regulator asks what the model was told and what it said on a given date, the answer exists.
PII redaction. Where policy requires, personal data is redacted before it reaches the model, and the redaction is logged too.
Guardrails on inputs and on retrieved content. Prompt-injection and topic guardrails run on what users type and on anything a retrieval pipeline hands the model, because a document in your own repository can carry an injection as easily as a user can.
Supply-chain integrity. Model files are verified by checksum against the publisher, mirrored internally and updated only through change control, so nobody pulls an unknown binary onto a production GPU.
Monitoring inside the perimeter. Latency, throughput, GPU utilisation and cost-per-request dashboards that live in the same network as the model, not in a SaaS observability tool.

All of it is written up as a control matrix against the framework that governs you — RBI's outsourcing and cloud expectations, SAMA's cybersecurity framework, DPDP, PDPL, or your sector regulator's — so your security team reviews a document before go-live rather than discovering gaps after. In our experience the control matrix is what shortens the approval cycle more than any technical choice.

What we bring from running regulated platforms

We should be precise here, because it is the kind of claim that gets inflated. The platforms below are not LLM deployments. They are the regulated, audited systems we already run, and the operating discipline a private model inherits from them is the point.

The Government of India's MOSPI platform coordinates national statistical surveys across states and districts on a custom Frappe application, deployed on secure government-compliant infrastructure with role-based access aligned to ministry hierarchy. It complies with Government of India data-residency and security requirements; its multi-level validation workflows cut data-quality errors by 75 %, and automated reporting took monthly report preparation from five days to same-day. The residency and role model are exactly what a ministry's private LLM would sit inside.

The Bank of Baroda early-warning system was rebuilt on Spring Boot, Angular and Liferay DXP 7.4 after the Vijaya Bank and Dena Bank mergers. It scaled to three times the data volume, implemented the maker-checker workflow RBI requires, brought risk-review cycles from two days to four hours, and migrated with zero data loss. Maker-checker and audit trails are the same controls a bank will ask for around a model that reads loan files.

The Royal Monetary Authority of Bhutan's EWS monitors the entire national banking sector: a multi-level maker-checker workflow with zero unauthorised indicator modifications, sub-second search across millions of financial records, and zero missed critical alerts for regulatory officers. A central bank is the strictest tier-one customer there is, and the habits it demands — change control, evidence, nobody editing a record alone — are the habits a private LLM deployment needs from day one.

None of those numbers came from a language model. They came from building systems that regulators inspect, which is the discipline we are describing.

Air-gapped: the extra work nobody budgets for

A closed network changes the project more than most plans allow for. Everything that is normally a download becomes a delivery. The model weights, the serving stack, the CUDA and driver versions, the Python packages for the applications on top, and the evaluation harness all have to arrive as a signed offline bundle, with versions pinned, and be installed by someone inside the perimeter who may not be your engineer.

Updates go through the same door. A new model version is a change-control ticket, a bundle, a checksum verification, an evaluation run inside the network against the same test set that chose the original model, and a rollback plan. That evaluation run is the piece teams most often leave out, and it is the reason a "routine update" can quietly make a production model worse with nobody able to say when. Keep the harness inside; run it on every change.

Monitoring lives inside too, which means the dashboards and alerting are part of the bundle, not a SaaS subscription. And the on-call arrangement has to work without remote access, which usually means a runbook detailed enough for the client's own operations team and a scheduled site visit rather than an SSH session. None of this is exotic; all of it takes time, which is why an air-gapped installation sits on the critical path of the timeline below.

The applications on top: the model is rarely the product

Nobody wants a model. They want the grounded assistant that answers policy questions with a citation, the extraction pipeline that turns scanned contracts into structured fields, the copilot inside the portal, or the agent that raises the ticket and drafts the reply. The private deployment gives those applications a place to run that the security team has already approved, which is why we build it first and build the applications on top.

The most common first application is a RAG assistant, and it sits on whichever tier you land on; the retrieval layer is the same. What changes on-prem is that the retrieval side also has to respect permissions — the model should only ever be handed pages the asking user is allowed to see — and that the embedding model and the vector store live inside the perimeter alongside the generator. How documents are chunked matters more than most teams expect; we wrote about why in our note on chunkless RAG, and the argument holds regardless of where the model runs.

Fine-tuning enters later, if at all. Owning the model means you can change it, and the temptation is to fine-tune early. Usually the wrong order: prompting and retrieval solve most enterprise needs, and both are reversible and cheap. When fine-tuning does earn its place — consistent structured output, house tone, domain vocabulary, or running a narrow task at a smaller size — we do it with LoRA or QLoRA on a curated set, where a few thousand high-quality examples beat a hundred thousand noisy ones, evaluate against a held-out set and the base model, and ship the adapter alongside the base so the base can be upgraded later. Training runs inside the same perimeter as inference; regulated data does not go out to be trained on and come back.

What it costs against the API

The cost question is arithmetic and we do it with you before recommending anything. On the API side the cost is tokens: your expected requests per day, times the average prompt and answer length, times the per-token price. On the private side it is GPU hours — bought or rented — plus the engineers who run them, divided by the requests you actually serve.

Below a certain volume the API wins. Above it, or as soon as data may not leave your control, the private deployment wins — and for the steady, high-volume workloads inside a bank or a government department the crossover comes sooner than most teams expect. Where the regulator has already ruled the API out, the comparison is moot and the cost model becomes a question of which tier-one option is cheapest: owned GPUs, a rented in-country rack, or a smaller model that fits the cards you already have.

We give you the model in a spreadsheet during discovery with your numbers in it, and we revisit it in a monthly operations report after go-live that compares what you are paying against the API alternative. That report is also the honest check on whether the deployment is still the right answer as prices and models change.

Timeline: what to expect and what stretches it

As a guide from our engagements: a two-week discovery produces the regulatory tier, the data-classification table, the task set, the model shortlist and a costed plan. A four- to six-week pilot stands up the chosen model on real hardware with your first application against it, and measures quality, latency and cost. Production — hardening, identity integration, audit logging, guardrails, monitoring, runbooks, the control matrix and hand-over — is typically two to four months from kick-off.

Two things stretch it, and both should be on the plan from the start: GPU procurement, which in this region can take longer than the build, and an air-gapped installation, for the reasons above. After go-live most clients keep us on a monthly retainer for model and driver updates through change control, capacity reviews and the compliance report; the engineers who built the deployment are the ones on call.

Where on-prem LLM projects go wrong

Buying GPUs before reading the regulator's text. The most common and most expensive error. The tier decision comes first; the hardware order comes after the pilot.
Applying tier one to all data. A classification table with a tier per data class is cheaper and just as compliant.
Choosing the model from a leaderboard. It was not benchmarked on your tasks, in your languages, at your precision.
Not measuring quantisation. The model that fits the card is a different model. Test it.
Audit logs as a phase-two item. They are the first thing the reviewer asks for and the hardest to retrofit.
Guardrails only on user input. Retrieved documents can carry injections too.
No evaluation harness inside the perimeter. Without one, nobody can say whether an update made the model worse.
No owner after go-live. GPUs, drivers and models all need updating; a deployment without an operations arrangement decays.

The short version

Read the regulator's text and classify your data before you touch hardware; most organisations need tier one for less than they think. Pick the model on a few hundred of your own prompts, at the precision you will actually run. Size GPUs from concurrency and token lengths, not a catalogue. Build the controls — isolation, identity, audit logs, redaction, guardrails, supply-chain integrity, monitoring — as the product, and hand the security team a control matrix before go-live. Build the applications on top once the platform is approved. Keep the evaluation harness inside the perimeter and run it on every change. Do the cost arithmetic honestly and revisit it monthly.

If you are weighing this for a bank, a ministry, a hospital group or a company whose contracts and drawings are the business itself, send us the data classes involved, your regulator, the applications you have in mind and rough request volumes. We come back within a week with the tier you are permitted, a model shortlist and a discovery proposal — and the rest of our enterprise AI practice is there for what you build on top.