Building an LLM feature, part 4: test and evaluate it

After part 3 we have a working categorizer. This part is about knowing how well it works, and noticing when a change makes it worse.

LLM features need two kinds of checks:

• Tests for everything deterministic: rules, normalization, batching, validation. They're fast, exact and run on every commit.
• Evals for the model: a fixed set of real cases, scored with metrics instead of pass/fail on each item.

Tests for the deterministic parts

The decision function from part 3 is pure, so it's easy to pin down with Vitest:

import { describe, expect, it } from "vitest";
import { chunk, decideWithoutLlm } from "./pipeline";

const staticPatterns = [
  { patterns: ["cuota de manejo"], category: "Comisiones bancarias", confidence: 0.92 },
];

describe("decideWithoutLlm", () => {
  it("prefers the user's learned rule over everything else", () => {
    const d = decideWithoutLlm(
      "CUOTA DE MANEJO TARJETA",
      [{ keyword: "cuota de manejo", category: "Tarjeta", matches: 2 }],
      staticPatterns,
      {},
    );
    expect(d).toMatchObject({ category: "Tarjeta", source: "learned" });
    expect(d.confidence).toBeCloseTo(0.86);
  });

  it("ignores accents and case", () => {
    const d = decideWithoutLlm("ÉXITO POBLADO", [], [], { Mercado: ["exito"] });
    expect(d.source).toBe("keyword");
    expect(d.category).toBe("Mercado");
  });

  it("leaves unknown merchants for the LLM step", () => {
    const d = decideWithoutLlm("PAGO PSE 88231", [], staticPatterns, {});
    expect(d).toMatchObject({ category: null, source: "none" });
  });
});

describe("chunk", () => {
  it("splits into batches of the given size", () => {
    expect(chunk([1, 2, 3, 4, 5], 2)).toEqual([[1, 2], [3, 4], [5]]);
  });
});

Note what these tests cover: policy (learned rules beat everything), normalization (accents and case), and the handoff to the model (unknown merchants come back as none).

Honest note: Nora's current tests for the AI categorizer re-implement helpers like chunking and accent removal inside the test file, instead of calling the real service with a fake client. That catches less than it looks. Pure functions like decideWithoutLlm are the fix: the code under test is the code in production.

A test for the model boundary, without the model

You can also test the service logic around the model with a fake client that returns a fixed answer. The things worth asserting:

• an index outside the batch is ignored,
• "Sin categoria" leaves the transaction untouched,
• a category that isn't in the user's list is never saved,
• each saved answer creates or updates a rule.

These tests don't tell you if the model is good. They tell you your code handles whatever it says.

The golden set

The golden set from part 1 is a JSON file in the repo:

[
  { "description": "RAPPI*DOMICILIOS", "expected": "Domicilios" },
  { "description": "UBER *TRIP", "expected": "Transporte" },
  { "description": "CUOTA DE MANEJO TARJETA", "expected": "Comisiones bancarias" },
  { "description": "PAGO PSE 88231", "expected": "Sin categoria" }
]

Rules for keeping it useful:

• Real descriptions only, from different banks and months.
• Include cases where "Sin categoria" is the right answer. Otherwise you'll reward a model that always guesses.
• Every production mistake someone reports becomes a new case.
• Test the model on descriptions your rules don't catch, because those are the only ones it sees in production.

The eval script

The eval runs the same categorizeBatch used in production and reports the metrics defined in part 1:

import { readFileSync } from "node:fs";
import OpenAI from "openai";
import { categorizeBatch } from "./categorize";
import { chunk } from "./pipeline";

interface GoldenCase {
  description: string;
  expected: string; // a category name, or "Sin categoria"
}

const golden: GoldenCase[] = JSON.parse(readFileSync("evals/golden.json", "utf8"));
const categories = [...new Set(golden.map((c) => c.expected))].filter((c) => c !== "Sin categoria");

async function main() {
  const client = new OpenAI();
  let correct = 0;
  let answered = 0;
  let wrong = 0;
  let inputTokens = 0;
  let outputTokens = 0;
  const failures: string[] = [];

  for (const batch of chunk(golden, 50)) {
    const res = await categorizeBatch(client, batch.map((c) => c.description), categories);
    inputTokens += res.inputTokens;
    outputTokens += res.outputTokens;

    batch.forEach((c, i) => {
      const got = res.byIndex.get(i) ?? "Sin categoria";
      if (got !== "Sin categoria") answered++;
      if (got === c.expected) correct++;
      else if (got !== "Sin categoria") {
        wrong++;
        failures.push(`${c.description}: expected ${c.expected}, got ${got}`);
      }
    });
  }

  const pct = (n: number) => `${((100 * n) / golden.length).toFixed(1)}%`;
  console.log(`cases:    ${golden.length}`);
  console.log(`accuracy: ${pct(correct)}`);
  console.log(`coverage: ${pct(answered)}  (answered with a real category)`);
  console.log(`wrong:    ${pct(wrong)}  (confident and incorrect: the number to watch)`);
  console.log(`tokens:   ${inputTokens} in / ${outputTokens} out`);
  failures.slice(0, 20).forEach((f) => console.log("  - " + f));

  // Fail CI if confident mistakes go above the agreed budget.
  if (wrong / golden.length > 0.03) process.exit(1);
}

main();

What the numbers mean:

• Accuracy: right answers, including correct "Sin categoria".
• Coverage: how often the model commits to a category.
• Wrong: confident and incorrect. This is the one with a budget. In this script the budget is 3% of cases; pick yours with the people who use the product.
• Tokens: so you notice when a prompt change doubles the cost.

The failure list matters as much as the totals. Reading twenty wrong answers usually tells you whether the problem is the prompt, a missing category, or a case that rules should handle.

Running it

• On every change to the prompt, the model or the batch size. Compare against the last run before merging.
• In CI, on a schedule or behind a label, since it calls a paid API. The script exits with an error when confident mistakes go over budget.
• Before upgrading a model version. A newer model isn't automatically better for your data.

When the eval fails

In order, what I check:

1. Is it a rule problem? If a common merchant keeps reaching the model, add a static pattern. That's cheaper and more reliable than any prompt.
2. Is a category missing or ambiguous? Two categories with overlapping meaning will confuse people and models alike.
3. Is the prompt missing context? For example, that descriptions are Colombian and amounts are in COP.
4. Only then, try another model or batch size.

In the last part we put it in production: logging, cost, retries, privacy and the loop that makes the model less necessary over time.