Running a language model on infrastructure you control is a legitimate architecture with clear trade-offs. It is worth deciding on the merits rather than on preference, because the economics are not subtle in either direction.
When self-hosting makes sense
- Data cannot leave your environment. This is the strongest reason and often the only one that matters.
- Volume is high and steady. Continuous heavy usage on a saturated accelerator can be cheaper than per-token pricing; sporadic usage almost never is.
- You need a specific model, a fine-tune of your own, or a version an API does not offer.
- Latency and locality requirements the API’s regions do not meet.
When it does not
- Traffic is bursty or low. You pay for the accelerator continuously and use it occasionally — see GPU instances bill for the instance, not the utilisation.
- Nobody on the team wants to own model upgrades, driver versions and serving stack maintenance.
- The requirement is a prototype. Prototype on an API and move when the volume justifies it.
Sizing: memory first
Serving capacity is decided by accelerator memory. Three things share it:
- The model weights, whose footprint depends on the parameter count and the numeric precision used.
- The attention cache, which grows with context length multiplied by concurrent requests — this is the term people forget, and it is what causes a server that worked in testing to fail under real concurrency.
- Activation memory during each forward pass.
Long contexts and high concurrency multiply together. Decide the maximum context you will actually serve and the concurrency you need, then size for that combination rather than for the model alone.
Throughput comes from batching
A single request leaves most of an accelerator idle. Continuous or dynamic batching — grouping concurrent requests into one pass — is where practical throughput comes from, and modern serving stacks do it for you. It also means throughput and latency trade against each other: larger batches serve more total tokens and make any individual response slower.
Decide which one the product needs before tuning.
What you are committing to
- Driver and runtime versions that must match the serving framework.
- Model artefacts stored somewhere durable, and a load time that is minutes rather than seconds on start.
- A health check that verifies the model actually responds, not merely that the process is running — see load balancers.
- Capacity planning, because there is no elastic pool behind you.
- Monitoring on accelerator memory, queue depth and time-to-first-token.
A workable pattern
- One GPU instance running the serving stack, with the model staged from object storage in the same region.
- A small CPU instance in front handling authentication, rate limiting and request queueing.
- A queue so that load in excess of capacity waits rather than failing.
- Scheduled shutdown outside serving hours, if the workload is internal and has none.
- A documented fallback to an API for overflow or outage.
That last item is worth having even if you never use it.
Where to go next
- Cloud accounts for AI training
- Choosing a cloud for AI workloads
- Cloud accounts with AI service access
Questions
Frequently asked
Is self-hosting a language model cheaper than using an API?
Only at high, steady utilisation. You pay for the accelerator continuously, so sporadic traffic is usually cheaper on per-token pricing. Data residency is often the stronger reason to self-host.
What decides how many concurrent requests I can serve?
Accelerator memory. The attention cache grows with context length times concurrency, so long contexts reduce how many simultaneous requests fit alongside the model weights.
Why is my self-hosted model slower than expected?
Usually because requests are being served one at a time. Continuous batching in the serving stack is what turns an underused accelerator into real throughput.