Building an LLM feature, part 3: build the rules, the model call and the service

In part 2 we designed the system. Now we write it, in three pieces:

1. A decision function that tries every non-AI option first.
2. A batched model call whose output is enforced by a schema.
3. A background service that ties them together and learns from each answer.

The examples are TypeScript. They type-check against the current openai and zod packages, and the tests in part 4 run against them.

1. Rules first

This is the heart of the ladder from part 1. It's a pure function: no database, no network, easy to test.

export interface Rule {
  keyword: string;
  category: string;
  matches: number;
}

export interface Decision {
  category: string | null;
  confidence: number;
  source: "learned" | "static" | "keyword" | "llm" | "none";
}

export const normalize = (s: string) =>
  s.normalize("NFD").replace(/[̀-ͯ]/g, "").toLowerCase().trim();

export function decideWithoutLlm(
  rawDescription: string,
  learned: Rule[],
  staticPatterns: { patterns: string[]; category: string; confidence: number }[],
  keywords: Record<string, string[]>,
): Decision {
  const text = normalize(rawDescription);

  const rule = learned.find((r) => text.includes(r.keyword));
  if (rule) {
    return {
      category: rule.category,
      confidence: Math.min(0.95, 0.8 + rule.matches * 0.03),
      source: "learned",
    };
  }

  const pattern = staticPatterns.find((p) => p.patterns.some((k) => text.includes(k)));
  if (pattern) return { category: pattern.category, confidence: pattern.confidence, source: "static" };

  for (const [category, words] of Object.entries(keywords)) {
    if (words.some((w) => text.includes(normalize(w)))) {
      return { category, confidence: 0.75, source: "keyword" };
    }
  }

  return { category: null, confidence: 0.1, source: "none" };
}

export function chunk<T>(items: T[], size: number): T[][] {
  const out: T[][] = [];
  for (let i = 0; i < items.length; i += size) out.push(items.slice(i, i + size));
  return out;
}

A few things worth noticing:

• Normalize once. Colombian descriptions come with and without accents and in any case. ÉXITO and exito must match.
• Order is policy. The user's learned rules come first because they encode their decisions. Static patterns (Nora has a file of common Colombian merchants and bank charges) come next, then keywords.
• Every decision says where it came from and how confident it is. That's what the confidence_score column stores.
• "No match" is an explicit result, not an exception. Those rows are the only ones the model will see.

2. The model call

What Nora's first version looks like

Nora's current code uses OpenAI's older function-calling API. This is a trimmed excerpt:

const res = await client.chat.completions.create({
  model: "gpt-4o-mini",
  messages: [
    { role: "system", content: systemPrompt },
    { role: "user", content: userMessage }, // "1. RAPPI*DOMICILIOS\n2. ..."
  ],
  functions: [buildCategorizationSpec()], // { description: string, category: string }[]
  function_call: { name: "categorize_transactions" },
  temperature: 0.1,
  max_tokens: 1000,
});
const call = res.choices[0]?.message?.function_call;
return call?.arguments ? JSON.parse(call.arguments) : null;

It works, and the service around it is careful: it ignores categories that aren't in the user's list and skips "Sin categoria". But it has two weak spots that I'd fix today:

• The model has to echo each description back, and the code matches answers by exact text. If the model changes a single character, that answer is lost.
• category is a free string. Validation happens only after the fact.

How I'd write it today

Structured outputs let the API enforce the schema, and a Zod enum restricts the category to the user's list:

import OpenAI from "openai";
import { zodResponseFormat } from "openai/helpers/zod";
import { z } from "zod";

export const UNCATEGORIZED = "Sin categoria";

const SYSTEM_PROMPT = [
  "You classify Colombian bank transaction descriptions into spending categories.",
  "Pick exactly one category from the allowed list for each numbered line.",
  `If none clearly applies, answer "${UNCATEGORIZED}".`,
].join(" ");

// The schema is built per request: the enum is the user's own category list.
export function buildSchema(categories: string[]) {
  const allowed: [string, ...string[]] = [UNCATEGORIZED, ...categories];
  return z.object({
    results: z.array(
      z.object({
        index: z.number(),
        category: z.enum(allowed),
      }),
    ),
  });
}

export interface BatchResult {
  byIndex: Map<number, string>;
  inputTokens: number;
  outputTokens: number;
}

export async function categorizeBatch(
  client: OpenAI,
  descriptions: string[],
  categories: string[],
): Promise<BatchResult> {
  const completion = await client.chat.completions.parse({
    model: "gpt-4o-mini",
    temperature: 0.1,
    messages: [
      { role: "system", content: SYSTEM_PROMPT },
      {
        role: "user",
        content: descriptions.map((d, i) => `${i}. ${d}`).join("\n"),
      },
    ],
    response_format: zodResponseFormat(buildSchema(categories), "categorizations"),
  });

  const byIndex = new Map<number, string>();
  for (const r of completion.choices[0]?.message.parsed?.results ?? []) {
    // Ignore indexes we never sent and explicit "no idea" answers.
    if (r.index < 0 || r.index >= descriptions.length) continue;
    if (r.category === UNCATEGORIZED) continue;
    byIndex.set(r.index, r.category);
  }

  return {
    byIndex,
    inputTokens: completion.usage?.prompt_tokens ?? 0,
    outputTokens: completion.usage?.completion_tokens ?? 0,
  };
}

Why each piece is there:

• buildSchema runs per request. The enum is the user's own categories plus "Sin categoria", so the model can't return anything else.
• index instead of text. Checking 0 <= index < descriptions.length is trivial and unambiguous.
• Low temperature. Classification wants consistency, not creativity.
• Tokens are returned with the result so the service can log cost per batch (part 5).
• The code still validates. Schema enforcement is a strong guarantee, but out-of-range indexes and "no idea" answers are handled explicitly anyway.

3. The background service

The service listens for statement.completed, finds what's still uncategorized, and processes it in batches. This is a simplified version of Nora's NestJS service, adapted to the new categorizeBatch:

@Injectable()
export class AiCategorizerService {
  private readonly logger = new Logger(AiCategorizerService.name);

  constructor(
    private readonly prisma: PrismaService,
    private readonly llm: LlmClientProvider, // returns null when no API key
  ) {}

  @OnEvent("statement.completed")
  async onStatementCompleted({ statementUploadId, userId }: StatementCompleted) {
    try {
      await this.categorize(statementUploadId, userId);
    } catch (err) {
      // Never let the AI step break the upload flow.
      this.logger.error(`AI categorization failed for ${statementUploadId}`, err);
    }
  }

  async categorize(statementUploadId: string, userId: string) {
    const client = this.llm.get();
    if (!client) return; // graceful degradation

    const pending = await this.prisma.transaction.findMany({
      where: { statement_upload_id: statementUploadId, category_id: null },
      select: { id: true, raw_description: true },
    });
    if (pending.length === 0) return;

    const categories = await this.prisma.category.findMany({
      where: { OR: [{ is_system: true }, { user_id: userId }] },
      select: { id: true, name: true },
    });
    const idByName = new Map(categories.map((c) => [c.name, c.id]));

    for (const batch of chunk(pending, 50)) {
      const result = await categorizeBatch(
        client,
        batch.map((t) => t.raw_description),
        categories.map((c) => c.name),
      );

      for (const [index, categoryName] of result.byIndex) {
        const tx = batch[index];
        const categoryId = idByName.get(categoryName);
        if (!categoryId) continue;

        await this.prisma.transaction.update({
          where: { id: tx.id },
          data: { category_id: categoryId, confidence_score: 0.85 },
        });
        // Upsert a CategoryRule (user, category, keyword) and bump match_count.
        await this.learnRule(userId, categoryId, normalize(tx.raw_description));
      }
    }
  }
}

The important behaviors:

• It only touches category_id: null rows. Anything a rule or a person already decided is left alone.
• Errors are caught at the edge. A failed categorization is logged; the statement is already saved and usable.
• Each answer upserts a CategoryRule, so the next statement with that merchant is solved in step 1, without the model.

One lesson from Nora: when a person corrects a category, Nora learns rules from up to three meaningful words in the description. When the model answers, it stores the whole normalized description as the keyword. Whole descriptions often include reference numbers (PAGO PSE 88231) and rarely match again, so word-level rules generalize better. It's on my list to unify.

What we have now

• Rules resolve everything they can, instantly, at upload time.
• The model sees only leftovers, in batches, with an output it can't get wrong in shape.
• Every answer is saved with its source and confidence, and turned into a rule.

That's a working feature. It isn't a trustworthy one until we can measure it. In part 4 we build the tests and the eval.