LLMs in production: the hard part is the contract around the model

For four years I led engineering at KitchenSync, an accounting automation platform for restaurants in the US. Every customer had its own chart of accounts and its own rules. Categorizing transactions and pulling data out of invoices was manual work, and when it went wrong nobody noticed until the month-end close, weeks later.

We put LLMs on that problem. The model call turned out to be the easy part. What took real engineering was everything around it.

1. Treat the model as an untrusted service

We integrated OpenAI, Anthropic and Gemini behind our own layer. The rest of the product never talked to a provider directly. That gave us three things for free:

• Fallbacks. If one provider times out or errors, the request goes to another one.
• One place to measure. Tokens in, tokens out, model, latency and cost are recorded for every call.
• Freedom to change models without touching product code.

2. Ask for structure, then verify it yourself

Every call asked for JSON that matched a schema. But "the model returned valid JSON" is not the same as "the answer is right", so after the model we ran deterministic checks that don't need AI at all:

• Does the account actually exist for this customer?
• Is the amount's sign what this kind of transaction should have?
• Is the date inside the period being closed?

If the output didn't parse, we retried once with the validation error in the prompt. If it failed again, it went to a person. We never accepted half-parsed output.

3. Let confidence decide who does the work

LLMs fail in three ways: the API fails, the format is wrong, or the answer looks plausible and is wrong. The first two are easy to catch. The third is the dangerous one.

Our answer was a confidence threshold. High-confidence results were posted automatically. Everything else went to a human review queue, with a Slack notification so it didn't sit there. Reviewers only looked at the uncertain cases instead of at everything.

4. Make every decision traceable

Each posting recorded who made the call: a rule, a model or a person, and for models, which prompt version. When an accountant asked "why is this here?", we had an answer. When we changed a prompt or a model, we could compare results before and after.

5. Cost is a design decision, not a bill

A few levers mattered more than any clever prompt:

• Right model per task. A small, cheap model for classification; a larger one only for hard extraction.
• Send less. Only the parts of a document the model needs, not the whole thing.
• Cache by a hash of the input, so the same document is never paid for twice.
• Batch anything a user isn't waiting for.
• Budget per customer, with alerts, because you can't control what you don't measure.

The feature was worth it because the cost per processed document was far below the cost of the accountant time it replaced.

What I'd tell a team starting today

Pick the model last. First write down the schema, the checks that prove an answer is usable, what happens when confidence is low, and how you'll know a prompt change made things worse. With that contract in place, swapping models is a configuration change. Without it, every model upgrade is a gamble.