Analysis: self-hosting an open-weight model — the arithmetic that decides it, and the costs nobody budgets for

The self-hosting conversation is unusually prone to bad arguments in both directions. One camp treats it as obvious that owning your infrastructure is cheaper, because the weights are free. The other treats it as obvious that nobody should run their own accelerators, because the hyperscalers have better utilisation than you ever will.

Both are describing the same curve from different ends. Hosted inference is a variable cost and self-hosting is a fixed cost, and every real decision is a question about where a specific workload sits relative to the crossing point.

The shape of the comparison

Write the two sides in the same units.

A hosted API charges per token. Ten million input tokens and two million output tokens a day cost a predictable amount, and the bill scales linearly with usage — down as well as up. On a quiet weekend, it falls.

Self-hosting charges per hour of accelerator, whether or not a request arrives. Rental prices for a current-generation datacentre GPU sit in the low single-digit dollars per hour per card from most providers, with wide variation by region, commitment and provider tier. Owned hardware converts that to capital expenditure plus power, cooling and depreciation, which usually lands in a similar range once amortised — the point is that both are per-hour, not per-token.

So the arithmetic is:

The consequence that catches teams out is the second row. A cluster sized for peak load and averaging 15% utilisation is paying peak-capacity prices around the clock for traffic that arrives in bursts. The token price comparison that looked favourable at full load is off by a factor of six at the real average.

Throughput is a software property

The most common analytical mistake is treating throughput as determined by the hardware. It is determined at least as much by the serving stack.

Batching dominates. Serving one request at a time on a large accelerator wastes most of it; continuous batching, where new requests join an in-flight batch rather than waiting for it to drain, can raise throughput by a large multiple on the same card. A self-hosting cost model built on single-request benchmarks will be wrong by that multiple.

Quantisation changes what fits. Serving weights at 8-bit or 4-bit precision cuts memory requirements substantially, which can move a model from two cards to one — and one card is not 50% cheaper than two, it is 50% cheaper and removes a whole class of interconnect complexity. Modern post-training quantisation methods are often close to lossless for many tasks, but "often" and "many" are load-bearing words: it has to be checked against your own eval set, because degradation is task-specific and an average score will hide it.

Model size is the lever with the largest effect. Most of the economic case for self-hosting depends on running a model small enough to serve many concurrent requests per accelerator. The open-weight ecosystem's steady improvement at small scale is precisely what made self-hosting viable for ordinary products; a task that genuinely requires a frontier-scale model has a very different hardware bill and a much weaker case.

The four lines missing from the first estimate

Every initial self-hosting estimate I have seen shares the same omissions.

Redundancy. One node is a demo. Production needs at least two, in different failure domains, plus headroom for a rolling restart. This is not a 10% adjustment; it is often a doubling of the hardware line before any traffic growth.

Engineering time. Someone configures the serving stack, tunes batch parameters, sets up autoscaling, writes the health checks, handles the driver upgrade that breaks the kernel, and is paged when throughput collapses at 2am. At any realistic loaded cost for that person, a fraction of one engineer's year is frequently larger than the entire GPU line for a mid-sized workload. This is the line most often left out, and it is often the one that decides.

Evaluation. Self-hosting usually means moving to a smaller model. That is a quality change, and confirming it is acceptable requires an eval set, a comparison run, and the judgement to interpret the result. It is real work, it happens before any savings arrive, and it recurs on every model upgrade.

The upgrade treadmill. Hosted APIs improve underneath you. A self-hosted stack improves when someone does the work: evaluating the new open release, re-tuning serving parameters, re-running the eval set, planning the migration. Budget it as recurring, because it is.

When self-hosting clearly wins

None of this is an argument against it. There is a specific profile where self-hosting is straightforwardly correct.

Steady, high, predictable volume. Batch classification, document processing pipelines, embedding generation, moderation queues — workloads that run continuously and keep accelerators busy. These are the best self-hosting candidates in existence, and the fact that many of them use small models makes the case stronger.

A task narrow enough for a small model. Classification, extraction, routing, structured tagging, translation within a fixed domain. If a well-chosen small model does the job at acceptable quality, the per-request economics against a frontier API are not close.

Offline or air-gapped operation. No API is available at any price. The decision is made elsewhere.

The reasons that are not about cost

Some of the strongest cases for self-hosting are indifferent to the arithmetic.

Data residency and regulatory constraint. Where data cannot leave a jurisdiction or a network boundary, self-hosting is often the only compliant architecture, and it stays correct even if it costs more.

A hard latency floor. Network round-trip to a hosted endpoint has a floor that local inference does not. For interactive experiences with tight budgets — voice, real-time assistance, in-editor completion — that floor can be the requirement rather than an optimisation.

Stability of behaviour. A hosted model can change under you; a pinned local checkpoint cannot. For systems that have been carefully tuned around a model's specific behaviour, that stability has value that does not appear on any invoice.

Independence from a vendor's roadmap. Deprecation schedules, rate limits, regional availability and terms of service are all decisions someone else makes. Owning the weights removes that dependency, at the cost of owning everything else.

The answer most teams land on

In practice, the common production outcome is neither pole. It is a routing arrangement: a small self-hosted model handles the large, well-understood, high-volume portion of traffic, and a hosted frontier model handles the hard tail — the ambiguous cases, the long documents, the requests where quality matters more than cost.

This captures most of the savings, because the high-volume portion is where the money is, while keeping a quality ceiling the small model cannot reach on its own. It also degrades gracefully: if the local cluster fails, traffic falls back to the API at a higher cost rather than to an outage.

The one thing worth insisting on is that the decision be made with the real numbers. Measure your own utilisation curve, benchmark throughput with batching enabled, price the engineering line at a real salary, and include the evaluation work. Do that and the answer usually becomes obvious. Skip it, and you will end up defending a position you adopted before you had any data.