AI Evals
How to test software whose answers are never exactly the same twice.
New to evals? Start with the Evals learning path: basics, writing rubrics, then tools.
Why evals exist
Evals are how you test software whose output is non-deterministic. Traditional software is deterministic: the same input gives the same output, so a unit test with a fixed expected value is enough. An LLM can give a different, equally valid answer every time, so pass or fail on an exact string breaks down.
Deterministic software
- Same input, same output
- Tested with unit tests and exact expected values
- Correct means matching the spec exactly
- Failure looks like a crash or a wrong value
Non-deterministic (AI) software
- Same input, different valid output
- Tested with evals: scored judgements against criteria
- Correct means good enough on the dimensions that matter
- Failure looks like drift, hallucination, tone or policy misses
The practical shift: quality becomes a score you track over time, not a green tick you get once.
Two types of evals
Model evals tell you which model to start with. Product evals tell you whether your product actually works for your users.
Model evals
- Question: which model is capable enough?
- Who runs them: model labs, benchmark publishers, and your team when shortlisting
- Examples: public benchmarks (MMLU, SWE-bench, GPQA) and leaderboards
- Use: a filter to narrow the field to 2 or 3 candidates
- Limit: generic. A high score says little about your use case
Product evals
- Question: does our feature do the job for our users?
- Who runs them: your product team, continuously
- Examples: tests built from your own user requests and failure modes
- Use: the ongoing quality gate for releases and prompt changes
- Limit: only as good as your dataset and rubrics
The common mistake is stopping at model evals. A model that tops a leaderboard can still fail on your domain, your data formats and your users’ phrasing. Product evals are where the real work, and the competitive advantage, sits.
Three ways to run product evals
Use all three in layers: cheap code checks on everything, an LLM judge for quality at scale, and humans to set and recalibrate the ground truth.
1. Human evaluation
- What: domain experts review outputs and label them good or bad
- Best for: setting ground truth; nuanced, high-stakes judgements
- Cost: slow and expensive; does not scale
- Watch out: reviewer disagreement; needs clear guidelines
2. Code-based checks
- What: deterministic tests written in code
- Best for: format and hard rules: schema checks, valid JSON, required fields, length limits, banned words; overlap scores such as BLEU against a reference
- Cost: fast, cheap, repeatable
- Watch out: cannot judge meaning. BLEU only counts word overlap, so a correct paraphrase can score near zero
3. LLM as judge
- What: a second model scores outputs against a written rubric
- Best for: subjective quality at scale: helpfulness, tone, faithfulness to sources
- Cost: fast and cheap once built
- Watch out: the judge can be wrong too; validate it against human labels
Writing rubrics
The skill that makes the LLM judge work is writing rubrics. Practitioners such as Hamel Husain recommend three rules:
- Binary, not scales. Pass or Fail, not “rate 1 to 5”. A binary call forces a clear definition and makes judge errors measurable.
- One judge per failure mode. “Does the answer use only facts in the source? Pass/Fail” beats one judge scoring “overall quality”.
- Show examples. Include at least one clear Pass, one clear Fail and one borderline case in the judge prompt.
Rule of thumb: if a check can be written in code, write it in code. Reserve the judge for what code cannot see, and reserve humans for defining what good looks like.
Building evals from a small dataset
Start small and real: a few dozen examples you have looked at closely beat thousands you have not. This is the workflow tools like Arize AI support.
User request
The actual input a user sends. Source: production logs, support tickets, or realistic synthetic cases.
Reference answer
What a good answer looks like. Source: written or approved by a domain expert. This is your ground truth.
AI answer
What the system actually produced. Source: your current prompt, model and pipeline.
The steps
Build the small reference dataset. Cover common requests, edge cases and known tricky inputs.
Do error analysis. Read the AI answers side by side with the references. Note every failure in plain words, then group them. Patterns emerge fast: “ignores the date range”, “invents a policy”, “too long for mobile”.
Name the evals. Each recurring failure pattern becomes a named eval, for example faithfulness, date handling, conciseness. You are measuring the failures you actually saw, not generic virtues.
Write the rubric. For each named eval, write a precise pass/fail definition, ideally with a passing and a failing example.
Run the judge. Apply the rubric with an LLM judge (or code, if the check is deterministic) across the dataset. Spot-check the judge’s verdicts against your own labels before trusting it.
The insight: evals come out of error analysis. Teams that write rubrics before reading real outputs end up measuring the wrong things.
How big, and how to trust the judge
Starter dataset
20 to 50 examples. Arize’s own guide starts here: enough to see patterns.
Error analysis
About 100 traces. Hamel Husain’s rough point where new failure types stop appearing.
Judge validation
About 50 human-labelled Pass and 50 Fail. Lets you measure how often the judge agrees with humans on each side.
Keep the reference answer column. Research on LLM judges found they struggle on questions they cannot answer themselves, and that giving the judge a good human-written reference answer closes much of that gap. Even a smaller judge with strong references agreed with humans better than a stronger judge without them.
Re-run error analysis after every big change: a new model, a prompt rewrite, a new feature or a production incident.
The eval loop
Evals are never finished. They run as a loop, and production is where the next test cases come from.
Simulate on cases. Run the system against the reference dataset before anything ships.
Build evals and score. Apply code checks, the LLM judge and human review to get a score per named eval.
Run in production. Ship, then sample live traffic and score it with the same evals.
Improve, fix, run again. Change the prompt, retrieval or model; add every new production failure to the reference dataset; go round again.
Each turn of the loop makes the dataset harder and the product better. Regressions get caught because yesterday’s failures are now permanent tests.
What this means for a product manager
- Evals are the spec. For AI features, the rubric is where product intent gets written down. Owning the rubric is owning quality.
- Error analysis is product work, not just engineering work. Reading real outputs and naming failure modes needs domain judgement.
- Model choice is a filter, not a strategy. Shortlist on benchmarks, decide on your own product evals.
- Layer the methods. Code checks for rules, LLM judge for quality, humans for ground truth and calibration.
- Ship the loop, not just the feature. A team without a working eval loop cannot tell whether a prompt change made things better or worse.
- Questions to ask any AI team: What is in your reference dataset? Which failure modes have you named? How often do you check the judge against humans? What happens when a production failure is found?
Key terms
Trace
The full record of one AI interaction: input, steps, tool calls, output.
Ground truth
The human-agreed correct answer or label you measure against.
Reference answer
An expert-written good answer stored in the dataset.
Error analysis
Reading outputs, noting each failure, then grouping notes into failure modes.
Failure mode
A named, recurring way the system goes wrong.
Rubric
The written Pass/Fail definition a judge applies.
LLM as judge
Using a model to score another model’s output against a rubric.
BLEU
A 2002 machine translation metric counting word overlap with a reference.
Benchmark
A public standard test used to compare models (MMLU, SWE-bench, GPQA).
Online eval
An eval run continuously on live production traffic.
Sources
- Arize: LLM as a judge
- Arize: Align LLM evals with human judgment
- Hamel Husain on error analysis and evals (How I AI recap)
- Hamel Husain on Lenny’s Podcast: evals, error analysis, better prompts
- Hamel Husain: eval audit checklist (evals-skills)
- Krumdick et al.: No Free Labels, limitations of LLM as a judge without human grounding
- A Practical Guide for Evaluating LLMs (BLEU and its limits)
- Weights & Biases: Beyond BLEU and ROUGE
Based on a working document of the author’s, fact-checked against the sources above. For terms, see the Jargon Decoder.