LLM Evaluation: How to Test AI Output Before You Ship It
Published 2026-09-07 · Updated 2026-09-07 · 6 min read · AI 工作術 (FreeCo Co., Ltd.)
LLM evaluation without an academic framework: a 30-case test set, blind review plus an LLM judge, and a regression gate on every prompt or model change.
"The new model feels smarter." On our team, that sentence is banned as a conclusion. Feelings can't ship, and feelings can't roll back. After a year-plus of running LLM features in production — and paying the model bills ourselves — here's the position we've landed on: an LLM application without an evaluation set is untested software. You don't know how good it is today, and you have no idea whether your next change makes it better or worse.
The good news: you don't need an academic benchmark framework, a research team, or an eval platform subscription. A spreadsheet with 30 well-chosen test cases is a real evaluation, and you can build it this afternoon. This article is the exact method we use — test set, scoring, regression gate — plus the three traps that quietly invalidate most homegrown evals.
Why "I tried it and it seems fine" isn't evaluation
LLM output is probabilistic. The same prompt is great today and mediocre tomorrow, and the three to five examples you try by hand are a tiny sample — usually of questions you already know well, which is exactly the bias that makes the conclusion worthless.
The bigger danger is the chain reaction of changes: you tweak the prompt to fix one failing case, and silently break ten others. Without a test set, you will not find out until users do. This is precisely the situation of writing code without tests and manually clicking through the app before each release — and the solution is the same: make verification automated and repeatable.
Step 1: Build your test set
A test set doesn't need to be big. Representative beats large. Here's how we build ours:
- Sample from real inputs. Pull real cases from production (anonymized), and stratify them by scenario: mostly common questions, plus edge cases, plus known landmines that have failed before, plus questions the system should refuse — out-of-scope requests and attempts to bait it into breaking rules. A few dozen cases is enough to start; one to two hundred is plenty for most products.
- Define "what a good answer looks like" per case. Not a golden answer for word-for-word comparison — LLMs phrase things differently every run. Write criteria instead: which points must be included, what must never appear (fabricated numbers, claims beyond the knowledge base), and any format or tone requirements.
- Keep collecting landmines. Every time production misbehaves, that case goes into the test set. This is the most valuable source of test cases you'll ever have — it converts every incident into a permanent regression guard.
Step 2: Score it — blind human review plus an LLM judge
Blind human comparison is your most trustworthy baseline. Take the outputs of two versions — old prompt vs new prompt, model A vs model B — strip the labels, shuffle the order, and have a reviewer pick the better one. The blinding is not optional: with labels visible, people unconsciously favor the version they just spent two days tuning. We know because we've caught ourselves doing it.
LLM-as-judge is how you scale. Write your criteria into a judging prompt and have another model score every case, so you can run the full set on every single change. But know its documented biases: judges prefer longer answers, prefer certain styles, and tend to favor outputs from their own model family. So our discipline is: the LLM judge runs daily regressions; blind human review decides the big calls — model swaps, major prompt rewrites — and we periodically sample both to check that the judge's scores haven't drifted away from human judgment.
And one rule that saves money and time: anything a program can check, don't ask a model to check. Valid JSON, banned words, length limits — rule-based checks are fast, exact, and free.
Step 3: Regression-gate every change
The whole process collapses into one sentence: any change — prompt, model version, retrieval strategy, parameters — runs the test set before it ships, and only goes out if the score holds or improves. A provider releasing a new model version and you switching providers to cut costs are the same event in this process: run the set, compare, then decide. (This gate is exactly what makes the model-downgrade tactics in LLM cost control safe instead of reckless.)
This isn't theory for us. Our AI video-clipping engine has one step where an LLM picks the highlight moments out of a long video — the single most judgment-heavy call in the pipeline. We maintain a labeled set of "here's where the highlight actually is" cases, and every adjustment to the selection prompt regresses against it. Without that set, "the picks feel pretty good" turns into "why does the AI keep choosing the wrong clips" about three months later — asked by a customer.
"It feels smarter" is chat. Engineering sounds like: out of 100 cases, the last version passed 74, this version passes 81.
Three traps that quietly invalidate your eval
- The test set leaks into the prompt. If you tune the prompt against the same cases you grade with, the score will look great and mean nothing. Keep tuning cases and validation cases separate, exactly like train/test splits.
- Measuring only accuracy, never refusal. "Did it correctly refuse what it shouldn't answer" matters as much as "did it correctly answer what it should" — especially in any compliance-sensitive domain. Your should-refuse cases are your hallucination guardrails under test; they belong in the same regression run.
- Too many metrics at once. Score correctness, completeness, tone, and format separately from day one and nobody will know which number matters. Start with a single primary metric — this case passes or fails — and split it into dimensions only when you genuinely need to.
The version you can start today
Open a spreadsheet. Column one: the input. Column two: the pass criteria. Every column after that: one version's output and a pass/fail mark. Thirty cases, twenty minutes of human review per change. That bare-bones setup already puts you ahead of most LLM applications in production, which shipped — and keep shipping — on vibes.
When the case count and the change frequency grow, automate it: script the runs, add the LLM judge for daily regressions, keep humans on the big decisions. The order matters — spreadsheet first, automation second. Teams that start by building an eval platform usually never finish; teams that start with 30 rows are regression-testing by Friday.
FAQ
Q: How big does an LLM evaluation test set need to be?
Smaller than you think. A few dozen representative cases is enough to start catching regressions, and one to two hundred covers most products well. Representativeness matters far more than volume — a stratified set with common cases, edge cases, past failures, and should-refuse cases beats a thousand random samples.
Q: Is LLM-as-a-judge reliable enough to use?
Reliable enough for daily regression runs, not reliable enough for big decisions. Judges have known biases — toward longer answers, certain styles, and their own model family — so use them for scale, keep blind human review for model swaps and major rewrites, and periodically audit the judge's scores against human scores to catch drift.
Q: How often should I run my evaluation set?
Before every change ships — prompt edits, model version bumps, retrieval tweaks, parameter changes all count. The set is a gate, not a quarterly report: if the score drops, the change doesn't go out. Cheap rule-based checks and an LLM judge make running the full set on every change practical.
Q: What should I do when a test case fails in production but passed in eval?
Add it to the test set immediately — production failures are the most valuable test cases you will ever collect. Each one becomes a permanent regression guard, so the same mistake can never ship twice. Over time this makes your eval set converge on the failure modes that actually matter for your product.