Three ways LLMs fail in production, and what to do about each
"The model might be wrong" is too vague to design for. In production, LLM features fail in three distinct ways, and each one needs its own defense. Treating them as one problem is how teams end up with retries that don't help and reviews that catch nothing.
Failure 1: the API fails
The provider times out, returns an error or rate-limits you. This is the easiest failure, because you know it happened.
What works:
- Short timeouts. A request that hangs for a minute is worse than one that fails in ten seconds and retries.
- Retries with backoff. Wait a little longer each time, so you don't hammer a struggling service.
- Provider fallback. If one provider is down, send the request to another. This is one of the best reasons to put every provider behind your own layer, and one of the reasons we integrated OpenAI, Anthropic and Gemini at KitchenSync.
- A circuit breaker. If a provider keeps failing, stop calling it for a while. One broken dependency shouldn't take the whole workflow down with it.
Failure 2: the output is broken
The API answers, but the output isn't usable: invalid JSON, a missing field, a value of the wrong type.
What works:
- Validate against a schema, always. Don't parse "most of it" and hope. Output that doesn't match the schema is a failure.
- Retry once, with the error. Send the validation error back to the model and ask it to fix its answer. This often works, and it's cheap.
- Then stop. If the second attempt fails too, send the item to a human queue. Looping retries rarely fix a broken prompt, and they cost money.
Failure 3: the answer looks right and isn't
This is the dangerous one. The JSON is valid, the fields are there, and the value is wrong. Nothing crashes, so nobody notices, until someone finds the error weeks later.
There's no single fix, so you stack defenses:
- A confidence threshold. High-confidence results go through automatically; everything else goes to review. Reviewers look at the doubtful cases instead of everything.
- Deterministic sanity checks. Rules that don't need AI: amounts with the wrong sign, dates outside the period, an account that doesn't exist. They're cheap, fast and they never hallucinate.
- Sampling. Even high-confidence results get a periodic human review of a random sample. That's how you find the errors your threshold is letting through.
- A golden dataset. A set of real inputs with known correct answers. Before any change to a prompt or a model goes to production, run it against this set and compare.
- Evals in CI. Make that comparison automatic, so a prompt change is tested like a code change.
Why the difference matters
Each defense only works for its own failure:
| Failure | You know it happened? | Main defense |
|---|---|---|
| API error | Yes | Timeouts, retries, fallback, circuit breaker |
| Broken output | Yes, if you validate | Schema, one retry, human queue |
| Plausible but wrong | No | Confidence, sanity checks, sampling, golden set, evals |
Retries do nothing for the third failure. A golden dataset does nothing for an outage. When you design an LLM feature, go through the three failures one by one and write down how you'll handle each.
For the broader picture of how these pieces fit into a product, see LLMs in production: the hard part is the contract around the model.