Why local-first models win
in regulated industries

The first meeting with a bank's risk team rarely involves model benchmarks. Someone asks where the customer data goes, and the answer decides whether the project continues.

With a hosted API the honest answer is: to a vendor, through their sub-processors, possibly to a region outside the country, under terms you did not negotiate and cannot audit. For a law firm handling privileged documents, a hospital holding patient records, or a lender processing credit data under the NDPA, that answer ends the conversation more often than it starts it.

What local actually means

The word covers three different setups, and people use them interchangeably.

Running the model on a GPU in your own data centre is the strict reading. Data never leaves the building, and the machine is yours to inspect.

Running it in a private cloud tenancy you control, in a region you chose, with no vendor access to the inference logs, is the practical reading. Most deployments we build land here.

Air-gapped means no outbound network at all. Updates arrive on media carried in by hand. This exists in defence and in parts of banking, and it constrains everything downstream, including how you patch a model.

Each has a different cost profile. All three answer the compliance question, and none of them answer it if you quietly send documents to a hosted API for the hard cases.

Where the hosted model breaks

Residency is the headline, but three other things bite later.

Sub-processor lists change. A vendor adds a region or a service and your data protection impact assessment is now out of date, which becomes your problem during an audit, not theirs.

Models get deprecated. A model you built a workflow around can be retired with a few months of notice, and the replacement behaves differently on your prompts. If your product depends on that workflow, you now have an unplanned migration.

Terms move. Pricing changes, rate limits tighten, and usage policies shift. A startup can absorb that. A bank with a three year technology plan cannot.

None of these are reasons to avoid hosted models in general. They are reasons why a regulated buyer asks for the local option, and why "just use the API" is not an answer a vendor can give them.

The quality gap, honestly

Two years ago the gap was large. Today it is narrower than most procurement documents assume, and it depends entirely on task.

For summarisation, extraction, classification and retrieval over a defined corpus, a well-chosen open model at seven to eight billion parameters does the job, especially after fine-tuning on your domain. We have seen small models outperform much larger general ones on narrow extraction tasks simply because they were trained on the right examples.

For open-ended reasoning across unfamiliar material, the frontier hosted models still lead. If that is the actual requirement, self-hosting will disappoint, and it is better to say so before a client spends on GPUs.

The useful framing: match the model to the task, not to the leaderboard. Most regulated workloads are narrower than the teams proposing them expect.

The cost curve nobody draws

The licence for an open model is free. The deployment is not.

You need GPU capacity sized for peak rather than average, because a chat interface that stalls at 10am is unusable. That means paying for idle headroom, or batching and accepting latency, or autoscaling on infrastructure that was not designed for it.

Then there is the operational load: driver versions, CUDA compatibility, model updates, monitoring, capacity planning, and someone on call when inference falls over at night. A hosted API turns all of that into a bill.

The honest comparison is not tokens against tokens. It is total cost of ownership against the value of the compliance position, and for many regulated buyers the compliance position is the product. We have seen a hospital accept a materially higher per-request cost because the alternative was not being allowed to run the project at all.

Fine-tuning is mostly data work

The LoRA adapters are the easy part. Training a small adapter on a single GPU takes hours, not weeks, and the tooling is mature.

The work is in the dataset. A few thousand clean, correctly labelled examples from your actual documents beat a hundred thousand scraped ones. Someone has to sit with the domain experts, define what a correct answer looks like, and adjudicate disagreements between them. In our experience that phase consumes most of the calendar time on these projects.

Two rules we hold to. Keep a held-out evaluation set from day one, drawn from real cases rather than generated ones. And never train on data that would be illegal to send to a hosted provider either, because the retention and access rules do not disappear when the model does.

Serving, in practice

Two serving stacks cover most needs.

vLLM handles concurrent users well, with paged attention and continuous batching, and it is the right default when several people or services share one model.

llama.cpp with a quantised GGUF file is the right choice when the target is a single workstation, a modest GPU, or a CPU-only fallback, and when you want a single binary you can copy onto a machine in a branch office.

Quantisation is the trade you make for memory. Four-bit quantisation cuts the memory footprint substantially and, in our testing on extraction and summarisation, costs little accuracy. Test it on your own evaluation set rather than trusting a general claim, because sensitive classification tasks degrade before fluent generation does.

Cap the context window deliberately. A large context is a memory bill and a latency bill, and most production tasks need far less than the maximum the model advertises.

Operations

Three things separate a demo from something you can leave running.

Version the model the way you version code. Keep the adapter, the base model hash, and the evaluation score for each release, and be able to roll back.

Log enough to debug, and nothing you could not defend in an audit. Prompt logs on regulated data are themselves personal data.

Build a fallback path for when inference is unavailable. A queue and a clear degraded state beat an error page, and it beats silently routing the request to a hosted API that your impact assessment does not cover.

When not to self-host

If the workload is genuinely general purpose, the volumes are low, and the data would be permissible to send out, a hosted API is cheaper and better. Say that plainly rather than selling infrastructure.

Self-hosting also makes little sense when nobody on the team can own a GPU server. A model with no one to operate it becomes an expensive experiment that gets switched off after the pilot.

A sensible starting point

Pick one narrow task with real volume: document summarisation, invoice extraction, first-line support triage. Build the evaluation set before choosing the model. Run the chosen open model and a hosted model side by side on that set, then decide with numbers rather than opinion.

If the local model clears the bar and the data is sensitive, deploy it for that task alone. Expand later. The first deployment teaches you what the second one costs.

We build these stacks, including the air-gapped variant, and we will tell you when a hosted API is the better answer for your case.