Schedules

Running evaluations on a cron, the API key a schedule runs under, and pausing one without losing its configuration.

A schedule runs one or more evaluations unattended. It is the difference between knowing a regression happened and finding out from a customer.

Field	Notes
Name	For you.
Cron expression	When to run.
Timezone	Which clock the expression is read in. Defaults to `UTC`.
API key	The credential the run executes under. See below.
Evaluations	One or more. A schedule and an evaluation are many-to-many.
Enabled	Whether it fires.

The timezone matters more than it looks

The cron expression is interpreted in the schedule's timezone, not the server's. 0 6 * * * with Europe/London is 6am London, which is 5am UTC in winter and 6am UTC in summer.

Set it to the timezone of whoever reads the results. A schedule that runs "overnight" in the server's timezone drifts into the working day twice a year.

The API key

A schedule holds an API key, because a scheduled run has no signed-in user to act as. The evaluations run with that key's identity.

The key cannot be deleted while a schedule references it. The reference is set to restrict deletion rather than cascade, deliberately: silently removing the schedule, or leaving it pointing at nothing, would both mean a suite that quietly stopped running. You will get a refusal naming the dependency instead.

Two consequences:

Rotating that key means updating the schedule, or it keeps the old one.
Tidying up credentials will hit this refusal, which is the system telling you something still depends on the key.

Several evaluations, one schedule

A schedule can hold several evaluations, and an evaluation can belong to several schedules. That pairing is what lets you run a fast suite hourly and the whole set nightly without duplicating any definitions.

Each evaluation in a schedule becomes its own run, so results stay per evaluation and a failure in one does not obscure another.

Pausing

Pausing stops a schedule firing and keeps everything else: its expression, its timezone, its key and its evaluations. Resuming puts it back without reconfiguration.

Prefer pausing to deleting while investigating something. A deleted schedule takes its evaluation pairings with it, and rebuilding the list from memory is how a suite ends up smaller than it was.

The API

GET    /api/evaluation-schedulers                      List schedules
POST   /api/evaluation-schedulers                      Create one
GET    /api/evaluation-schedulers/{id}                 One schedule
PUT    /api/evaluation-schedulers/{id}                 Update it
DELETE /api/evaluation-schedulers/{id}                 Delete it
GET    /api/evaluation-schedulers/{id}/evaluations     Which evaluations it runs
POST   /api/evaluation-schedulers/{id}/evaluations     Add one
DELETE /api/evaluation-schedulers/{id}/evaluations     Remove one
POST   /api/evaluation-schedulers/{id}/pause           Stop it firing
POST   /api/evaluation-schedulers/{id}/resume          Start it again

There is also a trigger endpoint the scheduler itself calls. It authenticates as the scheduling infrastructure rather than as a user, so it is not something to call by hand; to run a suite now, run the evaluation directly.

What to watch

A schedule that runs and always passes is telling you less than it appears to. The common causes are an expectation written so loosely that anything satisfies it, or a suite generated from the agent and never edited, which measures only whether the agent still behaves like itself. Test cases covers both.