Runs

What a run records, how cases are queued and retried, how progress is counted, and what settles a run whose cases never come back.

A run is one execution of every case in an evaluation. It carries a status, progress, the totals, the average score, and the agent it actually tested.

Status	Meaning
`running`	Cases are queued or in flight.
`completed`	Every case has a recorded result.
`failed`	The run could not finish. See stuck runs below.

What a run pins

A run records the agentId and the concrete resolvedAgentVersion it executed against, resolved once when the run is created.

This is not redundant with the evaluation. The evaluation's version is a selector: latest, published, or a number, defaulting to latest. So the evaluation says which agent to pick, not which one was picked, and both the selector and the agent id can be edited afterwards. Without the run recording what it resolved, editing an evaluation would silently rewrite the meaning of every past run.

It is also what lets results be matched to an exact agent version later, which is how the verdict shown beside an agent is calculated.

Runs recorded before this was stored carry neither field. They are still readable; they simply match no agent version, so they contribute to no verdict.

One task per case

Creating a run queues one task per test case, each carrying only the tenant, run, evaluation and case id. Everything else is looked up when the task runs.

That shape is deliberate, and it replaced a single detached promise inside one process:

A deploy or a scale-in abandoned the run.
Results were written only at the end, so every finished case was lost with it.
The row stayed running for ever.

With one task per case, the unit of work is something the queue can redeliver, time out and retry on its own, and a single wedged case costs one case instead of the run.

A case may take up to 600 seconds per attempt by default (EVALUATION_CASE_TIMEOUT_SECONDS). That is generous on purpose: a case invokes the agent and then has a model judge the answer, and forty seconds each is ordinary. It bounds one case, not the run.

Results are recorded once

Each result is keyed on the run and the test case together. A task the queue redelivers cannot produce a second row, so at-least-once delivery is safe to build on and a retried case does not inflate the totals.

Per case, a result holds the actual output, whether it passed, the score from 0 to 1, the judge's reason, and how long the case took.

Progress

Progress is the count of recorded results against the total, read from the database.

Two things follow. A reload does not lose it, and a run that dies halfway keeps every case it had already finished, visible as a run stuck below its total rather than as nothing at all.

A run can be watched live, or polled:

curl https://service.ensembleapp.dev/api/evaluations/progress/{runId} \
  -H 'Authorization: Bearer <jwt>'

Stuck runs

Cases are enqueued one at a time and best-effort. A case that cannot be queued is reported rather than rolled back, because the cases already queued are on their way and there is nothing useful to roll back to.

So a run can sit running with cases that will never arrive. A sweep settles those: it looks for running rows older than a cutoff and marks them failed. The cutoff defaults to four times the case timeout (EVALUATION_RUN_STALE_SECONDS), so around forty minutes with the default case timeout.

That means a genuinely slow run and an abandoned one look the same until the cutoff passes. A run marked failed with most of its cases recorded was almost certainly swept, not broken: the recorded results are still there and still accurate.

Run history

Runs are kept per evaluation, newest first, and a single run can be read back with all its results or deleted. Keeping history is what makes an evaluation useful over time: one run is a number, and the sequence is the thing that tells you whether a change helped.

GET    /api/evaluations/{evaluationId}/runs             Run history, newest first
GET    /api/evaluations/{evaluationId}/runs/{runId}     One run and its results
DELETE /api/evaluations/{evaluationId}/runs/{runId}     Delete a run
GET    /api/evaluations/progress/{runId}                Progress for a run in flight
GET    /api/evaluations/stream/{runId}                  Live updates as cases land