Five levers for LLM cost that matter more than a clever prompt

When a team adds an LLM feature, the first bill is usually a surprise. Not because the model is expensive per call, but because nobody decided how often it would be called, with how much text, and for whom.

At KitchenSync, our accounting platform for restaurants used OpenAI, Anthropic and Gemini, and cost was part of the design, not an afterthought. The question that justified the feature was simple: is the cost per processed document clearly below the cost of the accountant time it replaces? These are the five levers that kept the answer "yes".

1. Pick the model per task, not per product

"Which model do we use?" is the wrong question. The right one is "which model does this task need?"

Classifying a transaction or detecting a document's language is a small task. A small, cheap model does it well.
Extracting structured data from a messy invoice is harder. That's where a larger model earns its price.

Put every provider behind your own layer, so each task can point to a different model through configuration. Then "not everything goes through the most expensive model" becomes a setting, not a refactor.

2. Send fewer tokens

Most of the cost is text you didn't need to send.

Trim the context. If you only need the header and the line items, don't send the whole document.
Use few examples, well chosen. Two good examples usually beat ten average ones, and they're cheaper on every call.
Ask for compact output. A JSON object with short keys costs less than a paragraph, and it's easier to validate.

This lever also helps quality: less noise in the prompt means fewer chances for the model to focus on the wrong thing.

3. Never pay twice for the same answer

If the same document or the same question comes in again, the answer shouldn't cost anything.

Cache model responses by a hash of the input: the content plus everything that changes the result, like the prompt version and the model.
Cache embeddings the same way, since documents are often re-processed.

Including the prompt version in the key matters. When you change the prompt, you want fresh answers, not stale ones.

4. Batch what nobody is waiting for

Not every LLM call is interactive. If a user isn't looking at a spinner, the work can wait:

Group items into batches instead of one call per item.
Run them in off-peak windows.
Retry calmly, without the pressure of a user waiting.

Keep the interactive path for what really needs an immediate answer, and give it the tightest budget.

5. Measure every request

You can't control what you don't measure. For every call, store:

tokens in and tokens out,
which model answered,
the cost,
which customer and which feature it belongs to.

With that data you can set a budget per customer with alerts, spot one feature that suddenly costs ten times more than the rest, and compare models with real numbers instead of opinions.

Putting it together

None of these levers is advanced. Together, they turn LLM cost from a monthly surprise into a number you can predict and defend. If you're adding AI to a product, start with the measurement (lever 5): it tells you which of the other four to pull first.

The cost controls are one part of the contract around the model. I wrote about the rest (schemas, validation, confidence thresholds and traceability) in LLMs in production: the hard part is the contract around the model.