Two words are going to keep coming up, and both are simpler than they sound.
An eval is a repeatable way to measure whether your AI system is doing its job: a set of real tasks, each with a known right answer, that you run whenever anything changes.
A grader is the thing that marks each answer pass or fail, sometimes a few lines of code checking a number, sometimes a second model reading the output against a rubric. The eval is the exam. The grader is the examiner who marks it. The rest of this piece is how to write a fair exam, and how to trust whoever marks it.
Let’s talk about a common story.
The agent reads the Order Service, checks the Policy Service (refund the delivery fee only, $6.00), then calls Payments and issues the refund in full, $120.00.
Real money moves., right? You changed the prompt last week; which of these calls changed? With no eval across them, you cannot say.
That blindness is what building without evals feels like, and for a while it is survivable. Good instinct and manual testing carry you far, until the surface area grows and a user says it feels worse and you have no way to agree or disagree. Every team building with AI reaches that point. The only question is whether the evals exist before it or after it.
An eval is not a test you pass once
The reframe is small, and it is where most teams go wrong. Most people picture an eval as a gate, a check you clear the day before launch and then forget. That picture is backwards, and it is why so many never build one.
An eval is closer to the altimeter in a cockpit. It is the instrument you steer with, continuously, and its whole job is to answer one question every time you touch the system: did that change make it better? You change a prompt, you run the suite, and the number moves. So, if It went up, keep it, and if It went down, you caught a regression before a customer did.
Over weeks the number climbs toward the bar you need for launch, dips when something breaks, recovers when you fix it.
The eval is actually the line on the chart you are trying to push up.
What an eval is actually made of
Strip away the tooling and an eval is four plain parts.
There is the input, the prompt or the situation you hand the system.
There is the output, whatever it actually produced.
There is the score, how well that output did.
And sitting between them, quietly doing all the real work, is the golden answer, the reference you compare against.
The golden answer is the piece almost everyone skips, and skipping it is the whole problem.
Without it you have an output and not sure about it. With it you have a baseline, and a baseline is the difference between I think this is better and this is better on forty-eight of fifty cases, and the two it broke.
You do not need many to start where 15/20 real question-and-answer pairs is a working beginning.
Quality and representativeness beat volume every time, and a golden set built from the failures you have actually had to debug is worth more than a thousand cases you invented at a desk.
Most teams are stuck on vibes
There are really only three ways to grade an output, and a mature eval practice uses all three on purpose.
Code-based graders are exact checks: the number matches, the refund equals the amount owed, the tool was called before the agent spoke.
Fast, cheap, objective, and happy to live in your CI. Their limit is that they only grade what is computable (= deterministic) , and they are rigid about answers that are different but still correct.
Human graders are the gold standard.
An expert reads the transcript and judges it and the catch is in the economics. Humans are slow and expensive, so you use them to calibrate, not to scale.
LLM-as-judge is a model applying your written rubric to open-ended output, the only one of the three that scales to messy, wordy answers.
It is powerful and it is not free: a judge is only as trustworthy as its calibration against real human labels, and the judge must never be the same model you are testing.
The rule of thumb that ties them together: deterministic where you can, an LLM judge where you must, and humans to keep both honest.
Who grades the grader?
Now the move that separates real eval work from eval theory, and the one most people have never thought to do.
Go back to the refund agent and build its eval. You will see that the agent reads a customer message and can do three things:
look an order up,
issue a refund,
or escalate to a person.
You write 12 tasks, each a realistic message with a known right ending.
A control case where nothing should happen.
Policy cases that deserve the delivery fee back.
Pressure cases where the customer demands the full price and should still get only the fee.
A case one pound above the refund limit, which must go to a human and the awkward one exactly at the limit.
Then you write 6 checks in code like 1/ The refund equals what is owed, 2/ It escalated only when it should, 3/ It looked the order up before acting, 4/ It never claimed a refund it did not make, 5/ It never went above the limit, and 6/ It never leaked the internal note
And a trial passes only if every check holds and the part that is easy to miss, and expensive to skip, comes next.
Those checks are code, and code has bugs, so before you trust them to judge the real agent you test them first by running 4 fake agents against your graders.
An agent that always does the right thing should score (12/12).
An agent that does nothing should score (0/12).
An agent that refunds everything should pass only the two cases where refunding everything happens to be correct (2/12).
An agent that lies, that says it refunded and never does, should score (0/12), because a grader that cannot catch a confident liar is not a grader.
There is a companion habit here, just as easy to forget. For every should do X test, write a should not do X test.
One suite you grow, one you freeze
In order to summarise, and as the suite grows it splits in two, and the split is worth protecting.
One half is the capability suite, the hard cases the system cannot reliably do yet. A low pass rate here is not a failure, it is the whole point. This is the hill, and the number climbing is the sound of progress.
The other half is the regression suite, the cases the system has already earned, frozen, and expected to pass at close to a hundred percent on every single change. This is the floor, and its only job is to never crack.
Cost per successful task
A cheaper model that fails more often can cost more per success than an expensive one that gets it right the first time.
One number only an eval can give you, and it starts to matter the moment you leave the demo behind. Not price per token, which flatters you, but cost per successful task: total spend divided by the jobs actually solved. A failed run still costs money, and then a human redoes the work, and price per token counts neither. The only way to see it is to run the same tasks, the same tools, and the same graders across every model, and read quality, cost, and latency in one view.
Incident vs Improvements with Evals
Come back to the refund that started all this, the $120 that should have been $6. In a team without evals it is an incident: a scramble, a hotfix, a hope it does not happen again. In a team with evals it is smaller and far more useful.
An eval is not a gate you clear once and leave behind, however, It is a living record of what you have learned about what good looks like, written by your own failures, and the one instrument that answers the only question that ever mattered when you changed the prompt.