Evaluations
Score an agent against test cases you wrote, so a prompt change is a measurement rather than a hunch.
An evaluation is a set of test cases, each an input and what a good answer looks like. Run it and every case is sent to the agent, the answer is judged by a model against your expectation, and you get a score per case and a pass rate for the run.
It exists because agents cannot be regression-tested the way code can. There is no exact output to assert, so "did my prompt change break anything" has no answer until you have written down what good looks like.
The four objects
Evaluation
A named set of test cases, pointed at an agent, with optional shared context.
Test case
One input, one expectation, and how to judge the answer: a yes/no or a score with a floor.
Run
One execution of every case. Records the agent version it actually tested.
Schedule
A cron expression that runs one or more evaluations unattended.
What happens during a run
Four properties of this design are worth knowing before you rely on it.
Each case is its own queued task. A run is not a loop inside one process. A deploy or a
scale-in used to abandon a run mid-flight, lose every finished case and leave the row
saying running for ever. Now a single wedged case costs one case.
A result can only be recorded once. Results are keyed on the run and the test case together, so a task redelivered by the queue cannot double-count. That is what makes at-least-once delivery safe here.
Progress is a count of recorded results, not a number held in one pod's memory, so a run that dies halfway keeps the cases it had already finished.
The run pins the agent version it tested. An evaluation names which agent to pick,
possibly as latest, and both that selector and the agent id can be edited afterwards. The
run records the concrete version it resolved, so history cannot be silently rewritten.
A model judges the answers, so a score is a judgement and not a measurement. It is repeatable enough to catch a regression and not precise enough to argue about two points. Treat a run as evidence, not arithmetic.
Where to go next
Quickstart
Write an evaluation, generate cases, run it and read the results.
Test cases
Boolean versus scored cases, confidence floors, and generating cases from an agent.
Runs
The queue, progress, stuck runs, and what a run records about what it tested.
Schedules
Running evaluations on a cron, and the credential they run under.
Verdicts
How results become a pass, partial, fail or none, and when a verdict goes stale.