What is Inference?

The plain definition

Training fits the parameters. Inference uses them. Training is a project with an end date; inference is an operating cost that runs for as long as the product does.

That asymmetry is routinely mispriced. A fine-tuning run might cost a few thousand dollars once. Serving the resulting model to a hundred thousand daily users can cost more than that every month, indefinitely. When forecasting the economics of an AI feature, the inference bill is the number that matters.

The metrics that actually govern user experience

Latency is per-request time, and the average is close to useless. Report p50, p95, and p99 — the tail is what users notice and what breaks upstream timeouts. Throughput is requests served per second per unit of hardware, and it trades directly against latency: larger batches raise throughput and raise tail latency with it.

For generative models, two further numbers matter. Time to first token determines whether the interface feels responsive, since streaming output lets a user start reading while generation continues. Tokens per second after that determines whether reading feels comfortable. A model with mediocre total generation time but fast first token will usually feel better than the reverse.

Cost per thousand predictions is the metric to put in front of finance, because it makes the unit economics of a feature legible before it is scaled up.

How to make it cheaper and faster

Right-size the model. The most effective lever and the most often skipped. Distillation trains a small model to imitate a large one; task-specific small models frequently match a general large model on the narrow job you actually have, at a fraction of the cost.

Quantise. Serving weights at 8-bit or 4-bit precision instead of 16-bit cuts memory and increases speed, usually with minor accuracy loss. Measure the loss on your own evaluation set rather than trusting a published benchmark.

Cache. Real request traffic is heavily skewed. Exact-match caching handles repeats; semantic caching over embeddings handles near-duplicates. For language models, prompt caching over a shared system prompt avoids repeatedly paying to process the same prefix.

Batch what can wait. If predictions are not needed in real time, batch them. Overnight scoring of a customer base is dramatically cheaper per prediction than serving the same volume through a live endpoint.

Route by difficulty. Send easy cases to a small cheap model and escalate only uncertain ones to the expensive model. A well-tuned cascade often preserves most of the accuracy at a large fraction of the cost.

The operational parts people forget

Cold starts. Loading a multi-gigabyte model into GPU memory takes time. Scale-to-zero looks attractive on a cost sheet and can mean the first user after an idle period waits thirty seconds. Either keep a warm instance or accept that latency in the product design.

Training-serving skew. The classic silent failure: feature computation in the training pipeline and in the serving path are implemented separately, they diverge, and the model receives inputs at serving time that differ subtly from what it was trained on. Accuracy degrades with no error raised anywhere. A feature store exists largely to eliminate this by computing features once and sharing them across both paths.

Monitoring. Track input distributions and output distributions, not just uptime. A model that returns HTTP 200 for every request while its predictions have quietly degraded is the most expensive kind of outage, because nothing pages anyone. See model drift.