Evals and harnesses, in order

Two short paths for anyone building with AI. Each one starts with the basics and ends with something you can do on Monday. Read them in order, or jump to the step you need.

Path 1: Evals (how to test AI output)

Work through these in order. Each step builds on the one before.

1. 1What is an eval?The idea in plain words, with an everyday example.
2. 2Write your first evalHands-on, in a spreadsheet: pick a job, collect examples, write a pass/fail rule, score it.
3. 3Evals basicsThe two kinds of eval, and why you start with error analysis on real outputs.
4. 4Writing evals and rubricsTurning a failure into a pass/fail judge, and checking that judge against a human.
5. 5Using tools to build evalsAI assistants and platforms (Promptfoo, Braintrust, Langfuse) to speed up the work, without handing over the judgement.

Overview page: AI Evals.

Path 2: Harness (how to build the system around the model)

The model is only one part of a working AI product. These three articles cover the rest.

1. 1What is a harness?The idea in plain words, with a simple analogy.
2. 2Harness basicsWhat a harness is, the five parts it is made of, and why the same model can perform very differently.
3. 3Designing a harnessA step-by-step way to decide what context, tools, permissions and checks an agent needs.
4. 4Harness engineeringThe ongoing practice: turn every mistake into a fix in the harness, and keep the system improving.

Overview page: Harness Engineering.

How the two paths connect

Evals are how you know the harness works. Without evals, you are guessing. Without a harness, there is nothing consistent to evaluate. If you only have time for one path, do Evals step 1 and Harness step 1, then pick up the rest.

Compiled by Human Agency from the sources listed in each article.