Building an LLM feature, part 1: plan before you prompt

This series walks through one LLM feature from idea to production: how I planned it, how I designed it, the code, how I test it, and how I run it. The example is real. It's the transaction categorizer in Nora, a personal-finance app I'm building for Colombia (NestJS, Prisma, PostgreSQL, React Native).

The plan for the series:

1. Plan: the problem, what "good" means, and where the model belongs (this post).
2. Design: architecture, data model and the contract with the model.
3. Build: the rules, the LLM call with structured output, and wiring it into the app.
4. Evaluate: unit tests, a golden set and an eval that can fail CI.
5. Ship and operate: cost, failures, privacy and the feedback loop.

The code comes from Nora. Where I'd write something differently today, I show both versions and say which is which.

Start with the problem, not the model

Colombian banks don't give regular developers open APIs, so Nora's first feature is simple to describe: you upload your bank statement and your spending gets organized. That breaks down into two very different jobs:

• Reading the statement: turning a PDF, CSV or Excel file into rows with a date, an amount and a description.
• Categorizing each row: RAPPI*DOMICILIOS is food delivery, CUOTA DE MANEJO is a bank fee, PAGO PSE 88231 is… hard to say.

The tempting design is "send the whole PDF to a model and ask for categorized JSON". It demos well. It's also slow, expensive, hard to test, and it fails in ways you won't notice: a missing row, a swapped sign, an amount with the wrong thousands separator.

So the first planning question isn't "which model?". It's which parts of this problem actually need a model?

Where the model belongs

Walk down from the cheapest, most predictable tool to the most expensive one, and only move down when the step above has nothing to say:

1. Parsing is deterministic. Each bank's format is stable enough for a parser per bank and file type. Parsers are fast, free, and you can unit-test them against real files.
2. The user's own history comes first. If you already told Nora that a merchant is "Domicilios", that answer wins.
3. Known merchants and keywords cover a big part of everyday spending in Colombia without any AI.
4. The LLM handles the long tail: descriptions no rule recognizes.
5. When nobody is sure, leave it uncategorized. A person fixes it in two taps. A wrong category silently corrupts a budget.

This ordering is the most important decision in the whole feature, and it's made before writing a prompt.

Define "good" before writing code

For an LLM feature, "it works on my examples" is not a success criterion. I write these down first:

Criterion	What it means for the categorizer
Accuracy	Share of transactions that end up in the category a person would choose.
Coverage	Share that gets a real category instead of "uncategorized".
Confident and wrong	Share that gets a wrong category. This is the number to keep low, even if coverage drops.
Latency	The upload must feel instant. Categorization can finish a few seconds later.
Cost	Tokens per statement, and whether it goes down as users teach the system.
Privacy	What leaves our servers, and why.

Notice that coverage and accuracy pull against each other. A model forced to always answer covers everything and gets more wrong. The plan says, explicitly, that "uncategorized" is an acceptable answer and a wrong category is not.

Decide what the model is allowed to see

Financial data is sensitive, so this is a planning decision, not an implementation detail. The categorizer sends the model:

• the transaction descriptions that rules couldn't classify, and
• the names of the user's categories.

It doesn't need amounts, balances, dates, account numbers or the user's name to decide that UBER *TRIP is transport, so it doesn't get them.

Collect examples on day one

You'll need a golden set: real descriptions with the category a person chose. Start it before the feature exists:

• Take a few of your own statements from different banks.
• Categorize the rows by hand, including the ugly ones.
• Keep the file in the repo. It becomes your test suite in part 4.

The one-page plan

Before any code, the plan for this feature fit on one page:

• Problem: organize spending from uploaded statements.
• Non-AI parts: parsing, learned rules, known merchants, keywords.
• AI part: categorize leftover descriptions into the user's own categories.
• Rules of the game: uncategorized beats wrong; the model only sees descriptions and category names; the upload never waits for the model.
• Measures: accuracy, coverage, confident-and-wrong, tokens per statement.
• Data: a golden set from real statements.

In part 2 we turn this into an architecture, a data model and a strict contract with the model.