The LLM bill nobody forecast: seven levers for cutting inference cost

Why the forecast was wrong

The cost model that gets built during a pilot almost always underestimates production, and the reason is structural rather than careless. A pilot has a handful of users, short prompts, and no retries. Production has traffic skew, conversation histories that grow, retrieved context stuffed into every request, and retry logic that occasionally loops.

The number that matters is cost per thousand predictions at expected volume, and it should be calculated before the architecture is fixed rather than discovered on an invoice. See inference for why this dominates the lifetime cost of a machine learning system.

The levers below are ordered roughly by return on effort. The first two are where most of the savings usually are, and they are the two most often skipped.

1. Use a smaller model

The largest available model is the default choice and usually the wrong one. Most production tasks are narrow: classify this ticket, extract these fields, rewrite this into that format, decide whether this document is relevant. Narrow tasks rarely need frontier reasoning.

The way to find out is an evaluation set of real inputs with known good outputs. Run the small model against it. Teams are routinely surprised — a model a fraction of the size and price frequently matches the large one on the specific job, because the job never required the capability they were paying for.

Where the small model falls slightly short, light fine-tuning often closes the gap. LoRA adapters are cheap to train, produce a small artefact, and can be served against a shared base model. A fine-tuned small model beating a general large one on a narrow task at a fraction of the cost is a common outcome, not an optimistic one.

2. Cache, because traffic is skewed

Real request distributions have heavy heads. A meaningful share of queries to any support or search system are repeats or near-repeats.

Exact-match caching is trivial to implement and catches the literal repeats. Semantic caching catches near-duplicates by embedding the incoming query and returning a cached response when it is close enough to a previous one — set the similarity threshold conservatively, because a wrong cache hit is a wrong answer.

Prompt caching is the one most teams are leaving on the table. If every request begins with the same long system prompt, instructions, and few-shot examples, providers can cache that prefix so you are not paying to process it repeatedly. Structuring prompts so the stable part comes first and the variable part last is a small change with a direct effect on the bill.

3. Stop sending so much context

Attention cost grows with the square of sequence length, and you are billed on tokens. Long prompts are expensive twice.

The frequent offender is retrieval that returns too much. If you are putting 20 passages in the prompt because retrieval quality is uncertain, the fix is better retrieval — a reranker returning 5 good passages instead of 20 mediocre ones cuts token cost by roughly three quarters while improving answer quality, because the model is no longer distracted by irrelevant context.

Conversation history is the other. Sending an entire thread on every turn grows cost linearly with conversation length. Summarise older turns, or keep a rolling window plus a compact running summary.

4. Route by difficulty

Not every request needs the same model. A cascade sends everything to a small cheap model first, and escalates to the expensive one only when the small model's confidence is low or a validation check fails.

Tuned well, this preserves most of the quality at a substantial fraction of the cost, because the difficult minority is genuinely a minority. The engineering cost is a routing layer and a confidence signal you trust — which is why this comes after the simpler levers rather than before them.

5. Batch anything that can wait

Real-time serving is assumed far more often than it is needed. If a prediction is not being waited on by a user or a transaction, batch it — and the saving is commonly an order of magnitude, because hardware runs at high utilisation instead of sitting idle against peak capacity.

The test is simple: identify what changes between a scheduled run and the moment of use, and ask whether that change would produce a different decision. Risk scores, segment assignments, propensity models, and content classification usually pass. Providers also offer discounted asynchronous batch tiers for exactly this. See batch vs real-time inference.

6. Constrain the output

Output tokens typically cost several times more than input tokens, so verbose responses are expensive in a way that is easy to miss.

If you need structured data, use the provider's structured-output or JSON-schema mode rather than asking for JSON in prose — it is more reliable and it stops the model wrapping the answer in explanation you will discard. Set sensible maximum token limits. And where you have asked the model to reason step by step, check that you still need it; on current reasoning-tuned models, explicit chain-of-thought instructions often add tokens without adding accuracy.

7. Put a limit on the loop

The failure that produces genuinely alarming invoices is not steady traffic. It is an unbounded retry or agent loop meeting a long context — a request that fails validation, gets retried with the error appended, fails again, and repeats with a prompt that grows each time.

Every loop needs a hard iteration cap, a per-request token budget, and a timeout. Set spend alerts at the provider, not just at your cloud. And log token counts per request so a cost regression is visible in a dashboard rather than at the end of the month. If you are building anything agentic, this is not optional — see AI agents.

One last piece of sequencing advice: build the evaluation set before you start optimising. Every lever here trades something, and without a way to measure output quality you cannot tell whether a saving was free or whether you quietly shipped a worse product.