Writing evals and rubrics

A rubric is the written standard a judge uses to decide pass or fail. This article shows how to write one that people agree on, and how to check that it works.

Step 1: Pick one failure mode

Take a named failure from your error analysis, such as "quotes a price that is not in the source". Do not write one judge for "good answer" or "helpful". Those mean different things to different people, and the judge will guess.

Narrow is better. "Mentions a refund period not in the policy document" is an eval you can run. "Accurate" is not.

Step 2: Decide what can be checked by code

Before you reach for a model judge, ask whether plain code can do the check. It is cheaper, faster and always consistent.

• Output is valid JSON, or has the required fields → code.
• Reply is under 100 words, or contains no email address → code (regular expressions are fine).
• Answer uses the right product name or currency symbol → code.
• Answer is kind in tone, or explains a policy correctly → judge.

Use a model judge only for the checks that need reading comprehension.

Step 3: Write the rubric

A good rubric has four parts. Keep them short.

1. The question. One sentence, asked about one thing.
2. The definition of a fail. Concrete, observable, with an example.
3. The definition of a pass. The same, for the good case.
4. Edge cases. What to do when the answer is partly right, or the input is unclear.

Worked example

Failure mode: invents a refund policy

Question: Does the response state a refund period, deadline or
condition that does NOT appear in the policy text provided?

FAIL if the response states any refund rule (time limit, conditions,
exceptions) that is not supported by the policy text.
  e.g. policy says 30 days; response says "60 days".

PASS if every refund rule in the response appears in the policy text,
or the response says it cannot find the policy.

EDGE CASES
- Response paraphrases the policy correctly → PASS.
- Response says "please contact support" with no rule stated → PASS.
- Response refers to a refund rule that is ambiguous in the policy → FAIL
  unless it says it is unsure.

Answer with exactly one word: PASS or FAIL.

Notice what the rubric does not do. It does not ask for a score out of 10. It does not ask the judge to rate "overall quality". It asks one question and gives the judge the evidence it needs.

Why binary, not a score

Numeric scores drift. Is a 4 today the same as a 4 last week? Does a 4 from one judge equal a 4 from another? Pass/fail is easier to agree on and easier to count. When you need more detail, add a second binary eval for a second failure mode rather than making the scale finer.

Step 4: Check the judge against a human

A judge is itself a model, and it can be wrong. Before you trust its scores:

1. 1Hand-label about 50 examples.Read each output and mark it PASS or FAIL yourself, using only the rubric.
2. 2Run the judge on the same 50.Do not show it your labels.
3. 3Compare the two in a confusion table.Count the cases where you both said PASS, both said FAIL, and where you disagree in each direction.
4. 4Read the disagreements.Most of the time the rubric is unclear, not the judge. Fix the rubric, then test again.

Look at both kinds of error. A judge that misses real failures gives false confidence. A judge that flags good answers wastes your time and hides the real problems.

Sources

• Hamel Husain on LLM-as-a-judge and error analysis (hamel.dev)
• Krumdick et al., "No Free Labels": limits of LLM-as-a-judge without human grounding

Compiled by Human Agency from the sources listed. Method summarised from the sources above. For terms, see the Jargon Decoder.