Test cases
Boolean versus scored cases, the confidence floor, writing expectations a judge can apply, and generating cases from an agent.
A test case is an input, an expectation, and how to turn the agent's answer into a verdict.
| Field | Required | Notes |
|---|---|---|
input | yes | Sent to the agent as the user's message. |
expectedOutput | yes | What a good answer contains. Read by the judge, not compared literally. |
evaluationType | no | BOOLEAN or SCORE. Defaults to SCORE. |
minConfidence | no | 0 to 100. The floor a SCORE case must clear. Defaults to 70. |
description | no | For you. Why this case exists. |
The two kinds
BOOLEAN
The judge answers yes or no: does this answer satisfy the expectation. A pass stores a score of 1, a failure stores 0. The confidence floor does not apply, because there is no scale to apply it to.
Use it when the expectation is a fact that is either there or not. "States that the window is 30 days." "Declines to give legal advice." "Does not mention a competitor."
SCORE
The judge rates the answer from 0 to 100, which is stored as 0 to 1, and the case passes if
it reaches minConfidence. The default floor is 70.
Use it when an answer can be partly right, which is most of the time. "Explains the refund window, mentions proof of damage, and says how to start a claim" can be two thirds right, and a score says so where a yes/no would just say no.
Prefer BOOLEAN for anything you would argue about. A scored case near its floor flips between
runs for no reason you can act on, because the judge is a model. If a case matters, state the one
thing it must contain and make it boolean.
Writing an expectation the judge can apply
The expectation is read by a model and compared against the answer's meaning. That makes some phrasings work much better than others.
| Instead of | Write |
|---|---|
Your refund window is 30 days. | States the refund window is 30 days. |
30 | Gives the number of days as 30. |
A helpful, friendly answer. | Answers the question and offers the returns page link. |
Everything about refunds. | One case per claim, each naming its claim. |
Three rules that come out of that:
Describe content, not wording. An expectation written as the exact answer fails the moment the prompt is reworded, which is the change you most often want to make safely.
One claim per case. A case asserting four things tells you something broke and not which. Four cases tell you which.
State the absence too, when it matters. "Does not promise a refund without inspection" is a legitimate expectation, and the judge handles it, where a positive-only suite will never catch an agent that says too much.
Shared context
An evaluation can carry a data context: values every case can draw on. Use it for setup that is the same across cases, for example the account or region the whole suite is about, so each case's input stays the question rather than the question plus the scaffolding.
Generating cases
The platform will propose up to twenty cases from an agent's own prompt and tools, with an optional instruction to steer them.
It is worth using for a first pass, because an empty evaluation is the hardest one to start, and the generated cases show you the shape. But read them before keeping them:
- They describe what the agent would plausibly say, inferred from its prompt. That is not the same as what it should say, and the difference is exactly the bug you are looking for.
- An agent with a vague prompt generates vague expectations, which pass trivially.
- They tend to cover the happy path. The cases that catch regressions are the awkward ones, and those you still have to write.
A suite generated from the agent and never edited measures whether the agent still behaves like itself. That catches a real class of regression and will never tell you the agent was wrong from the start.
How many cases
Enough that a failure localises. A dozen specific cases beat fifty vague ones, since each case costs an agent invocation plus a judging call every run, and a suite nobody trusts gets ignored at exactly the moment it fires.