Monitoring
Call volume, error rates, slow operations and recent failures, for every API or one of them.
Every call to /external writes a row: which API and operation, the method and path, the
status, how long it took, which key made it and the request id. Monitoring is a read over
those rows, not extra instrumentation, so it covers every call ever made rather than
starting from when you turned something on.
Where to look
| View | Answers |
|---|---|
| API Endpoints → Activity | How is the whole external surface doing, and which API is noisy? |
| An API's Activity tab | How is this API doing, and which of its operations is the problem? |
Both take a window: the last 24 hours, 7 days or 30 days. Buckets are hourly up to a couple of days and daily beyond, because a month of hourly points is mostly noise on a chart a few hundred pixels wide.
What it shows
Volume over time, split by status class rather than exact status: successful, client
error and server error. The split is the useful one, because a rise in 4xx is a caller
that changed and a rise in 5xx is you.
Totals for the window: calls, errors, and the 95th percentile duration. The p95 is blank rather than zero when nothing was called, since an average of no calls is not zero.
A breakdown, one row per operation when you are looking at one API, or one row per API when you are looking at the whole surface, with calls, errors and p95 each. This is where a single slow operation inside an otherwise healthy API shows up.
Recent failures, each with its timestamp, method, path, status, error code, operation and the last characters of the calling key. Never the key itself.
The distinct-operations count comes from the database rather than from the length of the breakdown table, which is capped. An API with thirty operations called reports thirty, not twenty.
Reading a failure
The error code tells you whose problem it is, and the request id is how you find the rest.
| Seeing | Usually means |
|---|---|
VALIDATION_FAILED | A caller's request does not match the contract. The details in their response named the field. |
FORBIDDEN | A key without a grant, or a narrowed grant meeting an operation it does not name. |
OPERATION_NOT_FOUND | A path that does not match. Often a caller hard-coding a URL rather than reading the document. |
TARGET_FAILED | The tool or agent ran and reported a failure. The cause is in the tool, not here. |
TIMEOUT | The target did not finish in its budget. Either the budget is wrong or the target is. |
TARGET_UNRESOLVED | The definition points at something missing. A tool archived out from under a published API. |
INTERNAL | Ours. The request id is the only way to find what actually broke, since the caller was told nothing. |
Quote X-Request-Id from a caller's response, or the request id from a failure row, when
asking for help. It is the id in the server logs, the primary key of the row, and the id
an idempotency reservation was made under, deliberately all the same value.
Things the logs will not tell you
A response that did not match its declared output schema is logged as a warning, not recorded as an error, and the response was still sent. The call looks successful in monitoring, because it was. That warning is the only sign the contract and the target have drifted apart, so it is worth a search of its own if you declare output schemas.
A dropped log row. The call is recorded after the response has been sent, so logging never delays a caller and never fails a call. The cost is that a write failing leaves no row for a request that really happened.
Rate-limited requests that never authenticated. The failed-authentication budget is keyed on IP and refuses before the key is resolved, so there is no key or API to attribute those to.