CLIEncoders · AI & Agents
Private and on-premise AI deployment
For some organisations the constraint is not what a model can do but where the data may go. Self-hosting removes the question entirely — at the cost of owning the infrastructure.
When self-hosting is the right answer
Three situations make it compelling. Regulatory or contractual obligations that forbid sending data to a third party. Air-gapped environments with no outbound internet at all. And sustained high volume, where per-token API pricing eventually exceeds the cost of owning a GPU.
Outside those, hosted APIs are usually the better engineering decision — frontier capability, no operational burden, no capital outlay. We will tell you which category you are in rather than selling you the more involved option.
What we deploy
Open-weight models served with vLLM where throughput and concurrency matter, Ollama or llama.cpp where simplicity and modest hardware are the priority, and LocalAI where a drop-in OpenAI-compatible endpoint lets existing code work unchanged. Open WebUI or LM Studio where people need a usable interface rather than an API.
Around that: GPU sizing based on your real concurrency rather than a vendor's ideal benchmark, quantisation trade-offs measured rather than assumed, and the full private stack — self-hosted embeddings and a vector store alongside, since a private model calling a hosted embedding API defeats the purpose.
The honest trade-off
Open-weight models have closed much of the gap and are entirely sufficient for classification, extraction, summarisation and grounded question-answering. On the hardest reasoning tasks the frontier hosted models remain ahead.
You also take on operations: driver and CUDA versions, model updates, capacity planning, and being the one who fixes it at 3am. We size that burden honestly, including the option of a hybrid split where sensitive workloads stay local and the rest goes to an API.
Questions
What people ask before starting
What hardware do we need?
It follows from model size, how many people use it at once, and how long your prompts are — the last of which surprises people, since long-context workloads consume far more memory than the model weights alone suggest. We size it against your real usage pattern. Answers range from a single workstation GPU to a multi-GPU server, and getting this wrong in either direction is expensive.
Are open-weight models good enough?
For most production tasks, yes. Classification, extraction, summarisation, and RAG-grounded answering are all well served by current open models. For frontier reasoning on genuinely hard problems, hosted models still lead. The right way to settle it is your evaluation set on your data, which we build either way.
Is this cheaper than using an API?
Only above a certain sustained volume, and the crossover is higher than most people expect once you include hardware, power, and the engineering time to operate it. Below that, APIs win on pure cost. Above it, self-hosting wins and keeps winning. If your driver is compliance rather than cost, the calculation is different — you are buying a guarantee, and it is worth paying for.
Can this work fully air-gapped?
Yes. Models, dependencies and container images are staged and transferred in, with no outbound calls at runtime. It does mean updates become a deliberate procedure rather than a package command, so we document that path as part of the handover.
Related
Where this usually connects
Tell us what you are building
One technical call is usually enough to tell you whether this is straightforward, genuinely hard, or the wrong approach entirely. We would rather say so early than quote for the wrong thing.
Start the conversation