Nicolás Duque

Building an LLM feature, part 5: ship it and keep it healthy

Sep 14, 2026 · 4 min read

We have a planned, designed, built and evaluated categorizer (parts 1 to 4). This last part covers what happens after it ships.

The production loop

Production loop: a new statement arrives, rules answer known merchants, the LLM answers the rest, and answers and user corrections become rules for the next upload
The goal in production isn't only good answers. It's needing the model less each month.

Two sources feed the rules:

  • Model answers. Every category the model assigns is saved as a rule for that user.
  • User corrections. When someone changes a category, Nora marks the transaction as manual (automation never touches it again) and learns rules from the meaningful words in its description.

If the loop works, the share of transactions that reach the model should shrink for each active user. That's a metric worth watching, and it's the reason the design puts the model after the rules.

Degrade gracefully

The first production rule: the AI step is optional. In Nora:

  • With no API key configured, the categorizer returns immediately. Uploads, parsing and rules keep working.
  • Any error inside the AI step is caught and logged. The statement is already saved.
  • A failed batch leaves those transactions uncategorized, which the product already handles.

This makes local development, staging environments without keys, and provider outages boring, which is exactly what you want.

Timeouts and retries

Nora's first version doesn't configure these, and it should. The OpenAI SDK supports both on the client:

import OpenAI from "openai";

export const llm = new OpenAI({
  timeout: 20_000, // ms per request; a background job can wait, but not forever
  maxRetries: 2,   // retries connection errors, 429s and 5xx with backoff
});

Guidelines that have worked for me:

  • Keep timeouts short even in background jobs. A stuck request holds a worker.
  • Retry only what's transient. A schema or validation problem won't fix itself on retry.
  • Let failed batches stay uncategorized and pick them up on the next run instead of looping.
  • If you have more than one provider behind your own interface, fallback is a better answer to an outage than more retries.

Logs you can query

Nora's categorizer ends each run with one structured log line:

this.logger.log(JSON.stringify({
  event: "ai_categorization_completed",
  statement_id: statementUploadId,
  user_id: userId,
  total_uncategorized: uncategorized.length,
  total_categorized: totalCategorized,
  rules_created: totalRulesCreated,
}));

One JSON line per run is enough to answer the questions that matter:

  • How many transactions reach the model per statement, and is that going down?
  • How often does the model decline ("Sin categoria")?
  • Which users or banks produce the most leftovers? (Often a sign that a parser or a static pattern is missing.)

What I'd add: the token counts returned by categorizeBatch (part 3), the model name, and the duration. With those in the same line you can compute cost per statement without a separate system.

Cost

The design already does most of the cost work:

  • Rules first means most transactions never reach the model.
  • Batching spreads the fixed part of the prompt over 50 items.
  • A small model is enough for classification into a known list.
  • Learning rules turns each paid answer into free answers later.
  • Minimal input: descriptions and category names only, no statement dumps.

Then measure it: tokens per statement from the logs, and a budget alert per user or per day, so a bug that loops over the same batch shows up as an alert and not as an invoice.

Privacy

Financial data needs a clear answer to "what leaves our servers?":

  • The model receives transaction descriptions that rules couldn't classify and the user's category names. No amounts, balances, dates, account numbers or names.
  • Uploaded statements are stored encrypted (Nora's upload table keeps the encryption IV and tag next to the storage key).
  • Check your provider's data-retention and training settings, and document them for your users.
  • Don't log full prompts in production. The structured log above has counts and IDs, not content.

Watch the quality in production

The eval from part 4 protects you from regressions you can predict. Production shows you the rest:

  • Corrections are labels. Every time a user changes a category the model chose, you have a real mistake. Sample them and add them to the golden set.
  • Look at "Sin categoria" rows. If many share a merchant, that's a static pattern waiting to be written.
  • Re-run the eval before model upgrades, even "minor" ones.

Rollout checklist

  • The AI step can be turned off without breaking anything.
  • Timeouts and retries are configured.
  • Each run logs counts, tokens, model and duration.
  • There's a cost alert.
  • You can say exactly what data the model sees.
  • The golden set eval passed on the model version you're shipping.
  • User corrections are captured and reviewed.

Wrapping up the series

The model call in this feature is about fifty lines. Everything else (the ladder that keeps the model as a last resort, the schema, the idempotent data model, the rules that learn, the evals, the logs) is what makes those fifty lines safe to run on people's money.

If you're planning an LLM feature and want a second pair of eyes on the design, get in touch.