Write your first eval
You need a spreadsheet, about an hour, and one AI tool you already use. No code, no platform.
Step 1: Pick one job
Choose one task the AI does, not the whole product. Good examples:
- Answering questions about delivery times
- Summarising a meeting into action points
- Turning a customer complaint into a ticket with the right category
Write the job in one sentence: "The bot answers delivery questions using our delivery policy."
Step 2: Collect 20 real examples
Take real inputs from your past work: real emails, real meeting notes, real complaints. Remove names and personal details. Put each input in a row of the spreadsheet.
If you do not have real examples yet, write 20 realistic ones yourself. They are less useful, but enough to start.
Step 3: Get the AI's answers
Run each input through the AI and paste the answer into the next column. Do this once per version you want to compare. Keep the date and the version in the header, so you know which answers came from which setup.
Step 4: Write the pass/fail rule
This is the most important step. For the job, write down what a good answer must have, and what makes it a fail. Keep it to one or two rules.
Example rule for the delivery bot
PASS if the reply gives the delivery time that matches the policy, and does not promise a refund or a delivery date the policy does not allow.
FAIL otherwise.
Write it so a stranger could apply it the same way you would. If you disagree with yourself on two examples, the rule is not clear enough yet.
Step 5: Mark each answer
Go down the column and write PASS or FAIL next to each answer. Be strict and consistent. It takes about 10 minutes for 20 examples.
Look at the fails. Group them. Are most of them the same kind of mistake? That is the thing to fix first.
Step 6: Count, change, repeat
Work out the score: passes divided by total, as a percentage. Write it down with the version name.
Version Date Passed Score
v1 original 1 Nov 12 / 20 60%
v2 new prompt 8 Nov 17 / 20 85%Now change one thing, run the same 20 inputs again, mark them, and compare. Changing several things at once makes it impossible to know what helped.
Common first-timer mistakes
- Judging by feel. Write the rule before you look at the answers, or at least stick to it.
- A rule that needs judgement on every line. Split it into two simpler rules.
- Only easy examples. Include awkward and unclear inputs. They are where the failures hide.
- Changing the test when the score is bad. Fix the product, not the examples.
Once this works, what's next
When the spreadsheet gets too big or you need to run it every time you change something, you are ready for the next steps:
- Writing rubrics properly: sharper rules, and checking an AI judge against your own marking.
- Eval tools: when a spreadsheet is no longer enough.
Example figures are illustrative. For terms, see the Jargon Decoder.