Verdicts
How results become a pass, partial, fail or none beside an agent, why never-evaluated is not a failure, and when a verdict goes stale.
Runs produce numbers. A verdict is the one word shown beside an agent, most visibly in the sync preview when you are about to promote it to another tenant.
The four outcomes
| Verdict | When |
|---|---|
pass | Every test passed. |
partial | Some passed, some did not. |
fail | None passed. |
none | No tests cover this agent version at all. |
none is a first-class outcome, not an error, and it is the most common one. It must not render
as a failure: if "never evaluated" looked like "all tests failed", red would stop meaning anything
within a week. An agent with no evaluation is an unknown, which is a different thing to act on
than a broken one.
What a verdict is computed from
The latest completed run of each evaluation that covers the exact agent version, summed.
Three parts of that are load-bearing:
The exact version. Results are matched on the agent id and the concrete version a run
recorded. An evaluation pointed at latest is a selector, so it is the run's resolved
version that matches, never the evaluation's setting. See
Runs.
Each evaluation's latest run only. Re-running does not accumulate; it replaces that evaluation's contribution. So fixing a failure and re-running moves the verdict, rather than averaging the old failure in for ever.
Summed across evaluations. Three evaluations covering one version contribute their totals together, and the verdict reports how many evaluations took part, so "2 of 30 tests failed across 3 evaluations" is expressible rather than three separate verdicts to reconcile.
Staleness
A verdict can be marked stale, meaning the results predate the content they were run against, so the verdict describes an agent that no longer exists.
This only ever happens to a draft. A published version is immutable, so matching on the agent and version is a complete answer for it and there is nothing to go stale. A draft is edited in place, so an exact version match is necessary but not sufficient: the run may have tested what the draft said an hour ago.
A stale pass is the most misleading thing on the screen. It says the tests passed, and they did, against text that has since been edited. Re-run before promoting a draft, or promote the published version instead.
Staleness is decided by the query that fetches the summaries rather than in the browser, because both timestamps involved are stored without a timezone. Comparing them client-side would rest on each surviving the database driver and JSON serialisation with the same local interpretation, so it is done in one place, in the database, where there is no second implementation to drift.
Where verdicts appear
The sync preview. Promoting agents between tenants shows a verdict per agent, so pushing something whose tests are failing is a decision rather than an accident.
The agent list and the evaluation screens, as the summary of the latest results for what you are looking at.
A verdict can also be requested directly for a set of agents:
curl -X POST https://service.ensembleapp.dev/api/evaluations/agent-summaries \
-H 'Authorization: Bearer <jwt>' \
-H 'Content-Type: application/json' \
-d '{ "agents": [{ "agentId": "<id>", "version": 3 }] }'Each summary carries the totals, the pass count, how many evaluations contributed, when the most recent of those runs completed, and whether it is stale. The verdict word is derived from the totals, so a caller can show the same four outcomes without reimplementing the rule.
Reading one honestly
| Verdict | What it justifies |
|---|---|
pass, fresh | The cases you wrote still hold. Nothing about cases you did not write. |
pass, stale | Nothing yet. Re-run it. |
partial | Look at which cases. A known-acceptable failure is fine if it is known. |
fail | Either a real break, or the agent is erroring and every case is failing the same way. |
none | You have no evidence. Not the same as bad evidence. |
The ceiling on all of this is the suite. A verdict is a statement about the cases you wrote, judged by a model, and never a statement about the agent in general.