What is an eval?
An eval is a test you run on an AI system to check whether its answers are good enough. It is the AI version of a school exam: the same questions, marked the same way, so you can see whether things are getting better or worse.
Why AI needs evals
A normal calculator gives the same answer to 2 + 2 every time, so you can check it once and trust it. An AI can answer the same question in different words each time, and some of those answers are wrong in subtle ways. You cannot check it once and forget it.
Evals are how you keep checking, in a consistent way, as you change the prompt, the model, or the product.
An everyday example
Suppose you run a small online shop and use AI to answer customer emails about delivery.
- The questions: 30 real emails from the past month, such as "Where is my parcel?" and "Can I change my address after ordering?"
- The marking rule: each reply passes only if it gives the correct delivery time from the policy, and does not promise a refund that the policy does not allow.
- The score: 24 of 30 pass, so 80%.
Next month you change the prompt. You run the same 30 emails again. If the score goes to 90%, the change helped. If it drops to 60%, you know to undo it, and you know why.
The three parts of every eval
- 1QuestionsRealistic inputs. Start with 20 to 50.
- 2MarkingA rule that says pass or fail for each answer. It can be a simple code check, or a rule that another AI or a person applies.
- 3ScoreThe percentage that passed. Track it over time.
What an eval is not
- Not a one-off check. You run it again after every change.
- Not a vibe. "It seems better" is not an eval. A written pass/fail rule is.
- Not a public leaderboard. Public benchmarks help choose a model. Your own evals tell you whether your product works.
Want to try one yourself? Write your first eval in six steps. For terms, see the Jargon Decoder.