Private LLM deployment on your servers, under your rules

'Private LLM' covers three quite different setups, and the right one depends on what your regulator and your risk team require rather than on what sounds safest. The strictest tier is on-premise: open-weight models on GPUs in your own data centre or a rented rack in your country, with no path to the internet at all — the answer for defence, central banks, hospitals and anyone whose data may not leave a named facility. The second tier is your own cloud account: the same open models served on GPU instances inside your VPC in an approved region, with your keys, your logs and your network controls. The third is an in-region managed endpoint — Amazon Bedrock, Azure OpenAI or Google Vertex AI — where the model is the provider's but the contract guarantees no training on your data and residency in a region you choose.

Most enterprises we talk to assume they need the first tier and are permitted the second or third. The difference is significant: tier one means buying or renting GPUs, running them at useful utilisation and keeping a team on them; tier three means an API call and a contract. In discovery we read your regulator's actual text — the Reserve Bank of India's outsourcing and cloud guidance, SAMA's cybersecurity framework, DPDP, PDPL, or your internal policy — and write down which tier it permits for which data. Only then do we size anything. The RAG assistants we build sit on whichever tier you land on; the retrieval layer is the same.

The open-weight field is now good enough that the question is no longer whether a private model can do the job but which one does yours at the lowest cost. Llama, Qwen, Mistral, Gemma and DeepSeek families ship at sizes from a few billion parameters to several hundred billion; the smaller ones run on a single mid-range GPU and handle summarisation, extraction, classification and grounded question answering well; the larger ones approach the frontier APIs on reasoning and are worth their hardware for a narrower set of tasks. Multilingual quality varies sharply — Qwen is strong on Asian languages, some models handle Arabic and Hindi far better than others — so language is a selection criterion, not an afterthought.

We choose by running the candidates on a test set drawn from your real work: a few hundred prompts with expected outputs, scored by an LLM judge and spot-checked by your people. Quantisation is part of the test — a model quantised to fit a smaller GPU is a different model, and the quality loss has to be measured rather than assumed. Licences are checked before the download: most of these models are permissive for commercial use, a few carry conditions, and your legal team gets a one-page summary either way. The output of this phase is a decision you can defend: this model, at this precision, on this hardware, scoring this on your tasks, at this cost per thousand requests.

• Task set — real prompts and expected outputs from your workflows, in every language you need
• Candidates — three to five models and sizes, each at full precision and quantised
• Scoring — LLM judge plus human spot checks; latency and tokens per second measured on the target GPU
• Licence review — commercial terms, attribution and usage restrictions summarised for legal
• Decision record — model, precision, hardware, scores and cost per request, kept for the auditor

Production serving is vLLM in nearly every deployment we run. It batches requests continuously, manages GPU memory with paged attention so long contexts do not waste it, splits large models across GPUs with tensor parallelism, and exposes an OpenAI-compatible API — which means the applications, agents and RAG pipelines written against the API you are replacing move over with a base-URL change. Ollama is the right tool for a handful of users on a workstation, an edge device or a proof of concept; it is not what a bank's assistant should run on at month-end.

Sizing comes from arithmetic, not from a catalogue. Your expected concurrent users, the length of a typical prompt and answer, and the latency your users will tolerate give a required token throughput; the model, its precision and the GPU's memory and bandwidth give a delivered throughput per card; the ratio, with headroom for peaks, is the number of GPUs. As a rule of thumb, a quantised 7–9-billion-parameter model serves a department from a single 24 GB card; a 70-billion-parameter model wants two 80 GB cards or four 48 GB ones; anything larger is a cluster and a conversation about whether tier three is the better answer. We deploy on your hardware, on GPU instances in your cloud account, or on GPU capacity rented from data-centre providers in India and the Gulf, packaged with Docker or Kubernetes, behind NGINX with TLS, and — where the network is closed — installed from a signed offline bundle with no outbound access at all.

A private model earns its keep only if the controls around it are as strong as the isolation. The deployment runs in a segmented network with no outbound access from the inference hosts; access is through your identity provider with roles for who may call which model with which data; every request and response is logged with user, timestamp and model version to an audit store you control, with retention you set; PII is redacted before it reaches the model where policy requires; prompt-injection and topic guardrails run on inputs and on any retrieved content; and the model files themselves are verified by checksum against the publisher, mirrored internally and updated only through change control, so nobody pulls an unknown binary onto a production GPU.

We write this up against the framework that governs you — RBI's outsourcing and cloud expectations for Indian banks, SAMA's cybersecurity framework for Saudi financial institutions, the DPDP Act and Saudi PDPL for personal data, sector rules for healthcare — as a control matrix your security team can review before go-live rather than after. It is the same discipline behind the MOSPI platform, which runs under Government of India data-residency and security requirements, and the early-warning systems we operate for Bank of Baroda and the Royal Monetary Authority of Bhutan with their maker-checker and audit obligations.

Owning the model means you can change it, and the temptation is to fine-tune early. Usually the wrong order. Prompting and retrieval solve most enterprise needs — the model does not need to memorise your policies if a RAG pipeline hands it the right page — and both are reversible and cheap. Fine-tuning earns its place when the model must consistently produce a format (structured extraction into your schema), adopt a tone or house style, handle domain vocabulary it mangles, or run a narrow task at a smaller size than prompting would allow. It is poor at adding facts and worse at keeping them current.

When it is the right tool, we fine-tune with LoRA or QLoRA on a curated set drawn from your data — a few thousand high-quality examples beat a hundred thousand noisy ones — evaluate against a held-out set and the base model, and ship the adapter alongside the base so the base can be upgraded and the adapter re-trained rather than the whole thing frozen in time. Training runs on the same GPUs that serve, in off-peak windows, or on rented capacity for a day; the data never leaves the tier you chose.

The cost question is arithmetic and we do it with you before recommending anything. On the API side, the cost is tokens: your expected requests per day times the average prompt and answer length times the per-token price. On the private side it is GPU hours — bought or rented — plus the engineers who run them, divided by the requests you actually serve. Below a certain volume the API wins; above it, or as soon as data may not leave your control, the private deployment wins — and for the steady, high-volume workloads inside a bank or a government department the crossover comes sooner than most teams expect. We give you the model in a spreadsheet during discovery, with your numbers in it, and revisit it in the monthly operations report.

As a guide from our engagements: a two-week discovery produces the regulatory tier, the task set, the model shortlist and a costed plan; a four- to six-week pilot stands up the chosen model on real hardware with your first application against it and measured quality, latency and cost; and production — hardening, identity integration, audit logging, monitoring, runbooks and hand-over — is typically two to four months from kick-off, longer where hardware procurement or an air-gapped installation is involved. After go-live, most clients keep us on a monthly retainer for updates, capacity reviews and the compliance report; the engineers who built the deployment are the ones on call.

The organisations that need a private LLM are the ones whose data has a legal address: banks and insurers under RBI or SAMA, where customer and credit data stay inside approved infrastructure; government departments and public-sector companies with residency and security mandates; hospitals and healthcare groups with patient records; law firms, engineering consultancies and manufacturers with contracts, drawings and process knowledge that are the business itself; and companies in Saudi Arabia and the Gulf, where in-Kingdom hosting is increasingly the default expectation and where our Al-Khobar office delivers.

The model is rarely the product. It is the generation layer behind a grounded assistant, a document-extraction pipeline, a copilot inside a portal or an agent that raises the ticket and drafts the reply. The private deployment gives those applications a place to run that your security team has already approved, which is why we build it first and build the applications on top — the pattern across our enterprise AI practice.