Quickstart
Write an evaluation, generate test cases, run it, and read what came back.
1. Create an evaluation
Open Evaluations and create one.
| Field | Notes |
|---|---|
| Name | For you. Shown on the run history and in the sync preview. |
| Agent | Which agent the cases are sent to. |
| Agent version | latest, published, or a pinned number. See below. |
| User | Who the cases run as. The agent sees this user's context. |
| Data context | Optional values every case can use, for shared setup rather than per-case input. |
The version selector decides what you are testing. latest includes the draft, so it
measures what you are editing right now, which is what you want while iterating.
published measures what callers are actually getting. A pinned number measures one
revision for ever, which is how you compare two.
2. Add test cases
A case is an input and an expectation:
| Field | Example |
|---|---|
| Input | What is the refund window for a damaged item? |
| Expected output | Explains the 30-day window and mentions proof of damage. |
| Type | SCORE or BOOLEAN |
| Minimum confidence | For SCORE, the floor a case must clear to pass. Default 70. |
Write the expectation as what a good answer contains, not as the exact words. The judge compares meaning, so "mentions the 30-day window" survives a rewording of the agent's prompt and "Your refund window is 30 days." does not.
Generate test cases will propose up to twenty from the agent's own prompt and tools. Useful for the first pass, since an empty evaluation is the hardest one to start. Read them before keeping them: generated expectations describe what the agent would plausibly say, which is not the same as what it should say.
3. Run it
Running creates a run, resolves the agent version once, and queues one task per case. The progress bar is a count of recorded results, so it moves as cases land and survives a reload.
curl -X POST https://service.ensembleapp.dev/api/evaluations/run \
-H 'Authorization: Bearer <jwt>' \
-H 'Content-Type: application/json' \
-d '{ "evaluationId": "<id>" }'Cases run independently, so they finish out of order. A long run is normal: each case invokes the agent and then has a model judge the answer.
4. Read the results
Per case you get the actual output, whether it passed, a score from 0 to 1, and the judge's reason for that score. The run carries the pass count, the total and the average score.
The reason is the useful part. A score tells you something changed; the reason tells you what the judge thought was missing, which is usually either a real gap in the answer or an expectation that was written too strictly.
| Looking at | Usually means |
|---|---|
| A case failing with a reason you disagree with | The expectation needs rewording, not the agent |
| Several cases failing the same way | A real regression, most likely in the prompt |
| A case oscillating between runs | Its score sits near the floor. Raise the gap, not the floor |
| Every case failing at once | The agent itself is erroring. Check one case's output |
5. Keep it running
Once the evaluation is worth trusting, put it on a schedule so a regression is found by the clock rather than by a customer.
What to read next
- Test cases for how
BOOLEANandSCOREdiffer and where the confidence floor applies. - Runs for what a run records, and what happens to a case that never comes back.
- Verdicts for how runs become the pass or fail shown next to an agent.