# The REST profile grammar

> A profile is the shape of an API and carries no identity: a base URL template,
> the auth modes the API accepts, headers, the retry policy, where errors live in
> a non-2xx body, the window format, and the endpoints with their decoder and
> pager. An instance is one deployment of that shape: its credential kind and
> refs, `vars`, topic and interval, with per-unit narrowing. Every struct rejects
> unknown keys and every enum is a closed vocabulary, so a typo or an
> unsupported mode is a load error carrying the YAML line, never a setting that
> parses and does nothing.

The grammar as built is `crates/rest/src/profile/mod.rs`, and the generated
JSON Schema of the whole config surface (this grammar included) is
`docs/config-schema.yaml`. The shipped profiles under
`crates/fetcher/profiles/` are the worked examples of every entry below;
`config.example.yaml` shows an instance of one and an inline profile. Defaults
are not repeated here: read them off the `Default` impls in the grammar.

- [Vocabulary](#vocabulary)
- [Profile](#profile)
- [Auth](#auth)
- [Retry, errors and quota](#retry-errors-and-quota)
- [Window](#window)
- [Endpoints](#endpoints)
- [Rows](#rows)
- [Pagination](#pagination)
- [Constructs and prelude](#constructs-and-prelude)
- [Probe](#probe)
- [Templates](#templates)
- [Instance](#instance)
- [Validation](#validation)

## Vocabulary

Every closed enum of the grammar, with the shipped profiles that use each
entry.

| Axis | Vocabulary | Shipped profiles that use it |
|------|------------|------------------------------|
| Auth mode (`auth.accepts`, instance `auth.mode`) | `none` | pypi, crates_io, go_modules |
| | `bearer` (static token) | github, cloudflare, onepassword, slack, okta, runzero, gcp |
| | `api_key` (named header with a prefix, or a query parameter) | okta (`SSWS`) |
| | `basic` | none shipped |
| | `oauth2_client_credentials` (cached until shortly before expiry; per-unit `auth.scope`; `expose` allow-lists token-response fields for templates; `client_assertion` authenticates the exchange with a private-key JWT in place of a client secret) | azure, m365, bitwarden, crowdstrike, salesforce, runzero, okta |
| | `jwt_bearer` (RFC 7523 RS256 assertion from a service-account key, a key file or a bare PEM; claim templates) | gcp, gcp_pubsub, google_workspace, salesforce |
| | `gce_metadata` (the workload's token from the metadata server) | gcp, gcp_pubsub |
| | `signature` (a keyed digest over a canonical string of the request, in the digest, canonical shape and placement the profile names; Duo's versions are presets of it) | duo |
| | `sigv4` (AWS Signature v4, service and region rendered per request) | aws, object_store |
| | `credentials` (more than one credential on the one request, each where the profile places it; a placement may compose several into one value) | datadog |
| | `session_login` (a login call mints a session token, read from a JSON pointer or a response header and carried in a header of the appliance's naming) | none shipped |
| Pager (`paginate.strategy`) | `none` | runzero, azure `log_analytics` |
| | `link_header` (RFC 5988 `rel="next"`) | github, okta |
| | `cursor` (a token from the body or a header, injected into `query:`, `body:`, `body_replace:` or the path, and a list cursor is comma-joined) | aws, bitwarden, crowdstrike, datadog, duo, gcp, google_workspace, onepassword, slack |
| | `page_number` (bounded by `total_pages_at`) | cloudflare |
| | `offset` (advanced by rows received, bounded by `total_at` or a short page) | crowdstrike |
| | `request_path` (the next URL from the body or a header, relative resolved against the page) | azure, m365, salesforce |
| Decoder (`rows.decoder`) | `json_array` (streamed) | github, okta, m365 |
| | `ndjson` (streamed) | runzero, object_store |
| | `json_at` (a wrapped page, rows at a JSON pointer) | aws, azure, bitwarden, cloudflare, crowdstrike, datadog, duo, gcp, gcp_pubsub, google_workspace, onepassword, salesforce, slack |
| | `document` (the whole body is one row) | pypi, crates_io, go_modules, aws `cloudwatch_metrics` |
| | `json` (an array is its elements, anything else one row) | object_store |
| | `lines` (plain text, one row per line) | go_modules, object_store |
| | `csv` (one object per record, optional header) | salesforce `event_log_file` |
| Row flags (`rows.*`) | `gzip` (inflate a gzip member sequence before framing), `quoted` (each row is a JSON string holding the row), `header` (CSV) | object_store, aws `config`, salesforce |
| Construct (`construct.*`, at most one of the last three per unit) | `keyset` (one page sequence per key of a list, from a template or a first request) | aws, azure, pypi, crates_io, go_modules |
| | `lookup` (a second request per batch of ids the pages yield, with its own rows and pager) | aws `guardduty`, aws `cloudwatch_metrics`, crowdstrike |
| | `manifest` (a second request per item the pages point at; `key` and `position` give each row an item checkpoint) | m365, object_store, salesforce, go_modules |
| | `queue` (rows carry an ack id; the ack request goes out after delivery) | gcp_pubsub |
| `prelude` | idempotent requests sent once per tick before the first page | m365 (the subscription start) |
| Row builder (`rows.builder`, or `fold` for a folding one) | `columnar_table` | azure `log_analytics` |
| | `cloudwatch_metrics` | aws `cloudwatch_metrics` |
| | `go_module_aggregate` (a fold) | go_modules |
| | `wrap_non_object` | object_store |
| | `pubsub_message` | gcp_pubsub |
| | `cloudtrail_event` | aws `cloudtrail` |
| | `event_time` | azure `entra_signins`, `entra_directory_audits`, `entra_provisioning` |
| Lister (`lister`) | `s3` (`ListObjectsV2`, the keys modified after the unit's checkpoint) | object_store |
| Window (`window.format`) | `rfc3339_secs`, `rfc3339_millis`, `epoch_secs`, `epoch_millis`, `strftime:<pattern>`; `step` chunks the window, `lookback` is the window when the scheduler passes none; a unit may override the format | every incremental profile |
| Method | `GET` (retried by default), `POST` (retried only with `retry.retry_non_idempotent`) | every profile |
| Shape (`shape`, per profile or per endpoint) | `incremental` (rows are events in the window), `dump` (rows are the whole store, enveloped) | runzero is the shipped dump |
| Probe | the cheapest authenticated call, with an optional `fail_when`; without one the health check resolves the credential (and mints a token) | github, okta, cloudflare, duo, slack, onepassword, aws, pypi, crates_io, go_modules |

## Profile

| Field | Meaning |
|-------|---------|
| `profile` | Registry key of a shipped profile; an inline profile may leave it empty. |
| `maturity` | `alpha`, `beta` or `stable`; the fetcher warns at startup for an enabled non-stable source. |
| `base_url` | Template; must reference a variable, never a literal URL, because a profile carries no identity. Rendered per unit, so a unit's `vars` can pick the host. |
| `allow_hosts` | Origins a rendered URL may name besides `base_url`'s, for an API that answers on a second host (a sovereign cloud repointing `management_url`). Each entry is a template like `base_url`, so the host comes from the instance's vars, and names one exact origin: there are no wildcards, because a suffix match lets `example.com.evil.net` pass for `example.com`. Empty by default, and no shipped profile declares one. |
| `shape` | Default shape of every endpoint (`incremental` or `dump`). |
| `auth` | Accepted modes and the shape of each ([Auth](#auth)). |
| `headers` | Header templates on every request. |
| `retry`, `error`, `quota` | [Retry, errors and quota](#retry-errors-and-quota). |
| `window` | [Window](#window). |
| `probe` | The health-check request ([Probe](#probe)). |
| `defaults` | Values every endpoint inherits unless it sets its own ([Endpoints](#endpoints)). |
| `endpoints` | The units. |
| `vars` | Default `vars` an instance may override. |

## Auth

`auth.accepts` lists the modes an instance may pick; the sibling keys carry
the shape of each mode. Identity (tokens, keys, secrets) lives on the
instance, never here.

| Mode shape | Fields |
|------------|--------|
| `api_key` | `header` (the header carrying the key), `query` (the query parameter carrying it), `prefix` (text put in front of the key in a header, e.g. `"SSWS "`; a query parameter carries the key on its own, so a prefix beside `query` is refused rather than dropped). |
| `oauth2_client_credentials` | `token_url` (template), `scope` (omitted from the form when empty), `expires_in_fallback_secs` (lifetime assumed when the response carries no `expires_in`), `early_refresh_secs`, `expose` (top-level token-response fields the templates read as `auth.<name>`; `access_token`, `refresh_token`, `id_token` and `client_secret` are refused), `client_assertion` (see [Authenticating the exchange with a key instead of a secret](#authenticating-the-exchange-with-a-key-instead-of-a-secret)). The exchange posts `grant_type=client_credentials`, `client_id`, `client_secret` and the scope as a form. |
| `jwt_bearer` | `token_url` (template; may read `auth.token_uri` from a service-account key), `claims` (templates for `iss`, `scope`, `aud` and an optional `sub`; a claim that renders empty is left out), `ttl_secs` (`exp - iat`), `expires_in_fallback_secs`, `early_refresh_secs`, `expose`. The authenticator exposes `client_email` and `token_uri` from a service-account key and `token_url` once rendered. |
| `gce_metadata` | `url` (the service account's token URL on the metadata server, a template), `expires_in_fallback_secs`, `early_refresh_secs`. |
| `sigv4` | `service` and `region` (both templates, rendered per request from the unit's context; the body's SHA-256 is the payload hash). |
| `signature` | `preset` (a shipped scheme by name), or the scheme spelled out: `digest`, `keying`, `canonical`, `encoding`, `headers`, `signed_header_prefix`, `place`, `header`, `prefix`. See [Signing a request](#signing-a-request). |
| `credentials` | A list of placements: `header` or `query` (exactly one) with an optional `prefix`, carrying either `from` (one named credential as it resolved) or `value` (a value composed of `{{ credentials.<name> }}` placeholders). See [More than one credential on a request](#more-than-one-credential-on-a-request). |
| `session_login` | `url` (the login endpoint, a template), `send`, `username_field` / `password_field`, `body`, `token_at` or `token_header`, `ttl_secs`, `early_refresh_secs`, `header`, `prefix`. See [Logging in for a session token](#logging-in-for-a-session-token). |
| `bearer`, `basic`, `none` | No profile-side shape. |

A mode that mints a token for a scope (`oauth2_client_credentials`,
`jwt_bearer`) lets a unit ask for its own with `endpoints[].auth.scope`; units
with the same scope share one token.

### Authenticating the exchange with a key instead of a secret

A client secret is a static credential the deployment holds and the exchange
posts. Where a provider offers it, a key pair is the better trade: the private
half never leaves the deployment, and the grant it buys is scoped to the
integration rather than carrying the permissions of whoever minted a token.
Okta steers its API Services apps this way and calls it `private_key_jwt`;
Microsoft Entra, Zscaler OneAPI and Snowflake all take the same shape.

It is not a mode of its own. The exchange is still client credentials, so the
profile declares `client_assertion` inside the
`oauth2_client_credentials` block and the exchange posts
`grant_type=client_credentials`, `client_assertion_type` naming RFC 7523's
JWT-bearer assertion, the signed `client_assertion` and the scope, in place of
the `client_secret`:

```yaml
# profile
auth:
  accepts: [api_key, bearer, oauth2_client_credentials]
  oauth2_client_credentials:
    token_url: "{{ base_url }}/oauth2/v1/token"
    scope: okta.logs.read
    client_assertion:
      ttl_secs: 300
```

```yaml
# instance
auth:
  mode: oauth2_client_credentials
  client_id: "0oa1example"
  private_key: "vault:<mount>/data/<path>:<key>"
  private_key_id: "the kid the app holds the public half under"
```

| Field | Meaning |
|-------|---------|
| `ttl_secs` | Assertion lifetime, `exp - iat`. An hour at most -- a longer one is refused at load, because Okta answers `invalid_client` rather than shortening it. Short is right anyway: an assertion is spent the moment it is posted. |
| `claims` | Claim templates added to the assertion, or replacing `iss`, `sub` or `aud`. Each reads `vars`, `base_url` and `auth.token_url` / `auth.client_id`, and is held to the instance's context like every other credential template. The `expose` fields of the token response are NOT among them: they arrive with the token the assertion is minting. |

The assertion always carries `iss` and `sub` as the client id and `aud` as the
rendered token endpoint, which is what RFC 7523 s2.2 requires of a client
authenticating as itself; `iat` and `exp` come from `ttl_secs`, and `jti` is
fresh on every mint, because Okta refuses an assertion whose id it has already
seen. A profile that writes `iat`, `exp` or `jti` in `claims` is refused at
load rather than having them overwritten.

The signature is RS256, so `private_key` is an RSA private key PEM and a key of
another kind is refused when it is read, naming the kind that is wanted. The
instance's `private_key_id` becomes the `kid` of the JOSE header. An app holding
ONE registered key pair resolves an assertion without it; an app holding two --
which is what rotating a signing key looks like, the new public key registered
beside the old -- resolves by `kid` alone and refuses an assertion that names
none. So an instance whose app has ever been rotated carries it.

Declaring the block says the API accepts the path. Which path an instance takes
is decided by the credential it carries -- `client_secret` posts the secret,
`private_key` signs the assertion -- so one deployment of an API that takes both
may hold a secret and another a key pair. An instance carrying neither or both
is refused when it binds, as is a `private_key` against a profile that declares
no `client_assertion`.

### More than one credential on a request

Several prominent APIs authenticate a call with two credentials, and there is no
config workaround: a static added header is not a secret reference, and a key
does not go in plain config. The `credentials` mode is the shape for them. The
PROFILE declares where each credential goes and the name it is read under; the
INSTANCE supplies a spec per name, because a profile carries no identity. Each
name resolves once however many placements carry it, and every placement is
applied to the one built request.

`auth.credentials` is the mode for credentials the deployment supplies as specs
and the profile places as they resolved. Where a credential goes is an axis of a
request rather than a mode of its own, so a mode that MINTS carries its own
placement under its own key: `session_login` names the `header` and `prefix` its
token goes in, over a token it minted rather than over a spec.

Two credentials in two headers of their own -- Datadog, where the api key names
the organisation and the application key scopes the call to a user, so neither
can be dropped:

```yaml
# profile
auth:
  accepts: [credentials]
  credentials:
    - { header: DD-API-KEY, from: api_key }
    - { header: DD-APPLICATION-KEY, from: application_key }
```

```yaml
# instance
auth:
  mode: credentials
  credentials:
    api_key: "vault:<mount>/data/<path>:<key>"
    application_key: "vault:<mount>/data/<path>:<key>"
```

Two credentials composed into ONE header value -- Tenable Vulnerability
Management, whose `Authorization` carries both halves of the key pair, so the
single-header shape does not help. Tenable Security Center is the same shape
under `x-apikey` with the field names lower-cased:

```yaml
# profile
auth:
  accepts: [credentials]
  credentials:
    - header: Authorization
      value: "accessKey={{ credentials.access_key }};secretKey={{ credentials.secret_key }}"
```

```yaml
# instance
auth:
  mode: credentials
  credentials:
    access_key: "vault:<mount>/data/<path>:<key>"
    secret_key: "vault:<mount>/data/<path>:<key>"
```

One credential goes in as `from` with a `prefix` when the API wants text in
front of it (`{ header: Authorization, from: token, prefix: "Token " }`). A
`value` naming one credential says the same thing the long way round, so keep
`value` for the several a provider wants in the one place.

A name is letters, digits, `_` and `-` -- in the profile's `from`, in a
`{{ credentials.<name> }}` placeholder and as an instance key alike, because the
two sides are matched against each other. A spec is any the resolver reads; the
spellings are in `config.example.yaml`.

A `value` is composed, not computed. Its only substitution is
`{{ credentials.<name> }}`: anything else between the braces is refused at load,
so no expression ever runs over a secret and no context -- `vars`, `unit`,
`window` -- can reach a credential value. That is also why a composed value needs
no per-unit check of its own: there is nothing in it a unit could change. The
composed string is built once, on first use, and the placement that writes it
marks the header sensitive, as it does a single credential.

Refused at load, each naming the field it sits on -- a placement by its index:

- a placement that names neither or both of `header` and `query`, or neither or
  both of `from` and `value`
- a `header` that is not a header name, an empty `query` or `from`, or a `from`
  that is not a credential name
- two placements writing the one header, where the second would replace the
  first, or the one query parameter, where both would be sent
- a `query` the profile writes itself -- `defaults.query`, an endpoint's `query`,
  the pager's `into: query:<name>` or its `param` -- or that a unit's narrowing
  writes: a query pair is appended, so the credential would be sent twice
- a `value` in a `query`: a composed value carries more than one secret and a
  query parameter is written into every access log and proxy on the path, so it
  goes in a header. A single credential may still go in a query.
- a `prefix` beside a `value`, which says the whole header value already, or
  beside a `query`, which carries the credential on its own
- a `prefix` or a `value`'s literal text carrying what no header value may carry
  -- a newline, a tab -- checked here because the text is the profile's and
  static, so it is named with its field rather than failing on every tick
- a name the profile places and the instance supplies no spec for, and a name
  the instance supplies that no placement reads -- so a misspelling is refused
  from both ends rather than authenticating as half a credential

### Logging in for a session token

An appliance rarely has an OAuth2 endpoint. It has a login call: post the
administrator's credentials, get a token back, send that token on every later
request. vCenter, Cisco Catalyst Centre, Check Point Management, F5 BIG-IP,
Wazuh, Veeam, Zscaler's legacy API, Redfish and iDRAC BMCs and Qualys all work
this way, and none of them speaks RFC 6749: the token comes back under a name
the vendor chose, in the body or in a header, usually with nothing said about
how long it lives.

So `session_login` reads the token from where the profile points, holds it for
the lifetime the profile declares, and carries it in the header the profile
names. It is NOT a bearer, and the grammar does not pretend otherwise.

| Field | Meaning |
|-------|---------|
| `url` | The login endpoint, a template (may read `base_url`). |
| `send` | `body` puts the credentials in the JSON body, `basic` in the login's own `Authorization: Basic` header. |
| `username_field`, `password_field` | The body members the credentials are written into under `send: body`; `username` and `password` unless the appliance names them otherwise. |
| `body` | Further members of the login body, each a template (F5's `loginProviderName`). The profile's own text -- a credential never travels here. Refused under `send: basic`, which posts no body at all. |
| `token_at` | JSON pointer to the token in the login response. The empty pointer is the whole body, which is what vCenter answers: a JSON string that IS the session id. |
| `token_header` | The response header carrying the token instead (F5's `X-F5-Auth-Token`). Exactly one of this and `token_at`. |
| `ttl_secs` | How long the token is held. The appliance does not say, so this is the session lifetime it documents. |
| `early_refresh_secs` | How long before that lifetime is up the login runs again, with the same meaning it has on the OAuth2 spec: capped at half the lifetime. |
| `header` | The request header every later request carries the token in. Required -- there is no default, because there is no convention. |
| `prefix` | Text in front of the token in that header, for the appliances that want one. |

The instance supplies `username` and `password`, the same two fields the `basic`
mode takes.

```yaml
# profile
auth:
  accepts: [session_login]
  session_login:
    url: "{{ base_url }}/mgmt/shared/authn/login"
    send: body
    body: { loginProviderName: tmos }
    token_at: /token/token
    ttl_secs: 1200
    header: X-F5-Auth-Token
```

```yaml
# instance
auth:
  mode: session_login
  username: "svc-dfe"
  password: "vault:<mount>/data/<path>:<key>"
```

One login per instance: `url` and every `body` member render against the
INSTANCE's context, and one that reads a name only a unit supplies is refused
when the instance binds, as a token endpoint's is. A refused login carries its
status and nothing of the response text -- a login body holds the password, and
an appliance that answers a bad request by quoting what it was posted would put
that password in the one line every consumer logs.

`url` is held to the same rule as a token endpoint: https, or plain http on a
loopback address so a fixture needs no certificate. Plain http anywhere else is
refused however private the network, and so is any other scheme. An appliance on
a management LAN is exactly where a captured administrator password is worth the
most, and every appliance here serves https -- with its own certificate, which is
a trust-store question and not a reason to send the password in the clear. The
refusal is at BIND rather than at the first login, because the URL is rendered
there.

A login the appliance REFUSES is held for a few minutes and answered from rather
than posted again. vCenter, F5 and Check Point all lock an account after a
handful of failed logins, so a wrong or rotated password must not post the
administrator's credentials once per unit per tick for as long as the deployment
runs. An appliance that could not be reached is tried again by the next request,
because that is not a failure the account is locked for.

### Signing a request

Some providers authenticate a call with a digest of the call itself rather than
with a credential the request carries. The digest algorithm, the canonical
string it is taken over and where the digest goes are per-vendor details, so
they are the `signature` mode's config rather than a mode of their own.

| Field | Meaning |
|-------|---------|
| `preset` | A shipped scheme by name: `duo_v5` or `duo_v2`. A preset is the WHOLE scheme, so a block that names one and also sets another key is refused. |
| `digest` | `sha1`, `sha256` or `sha512`. The same digest takes the body hash and the header hash. `sha1` is there for Duo's legacy scheme and is never a default. |
| `keying` | `hmac` (the secret keys an HMAC over the canonical string, RFC 2104) or `prefix` (the secret is hashed in front of the canonical string, with no HMAC). |
| `canonical` | The canonical string, ONE LINE per entry, joined by newlines. Each is a template over `signature.*` and nothing else. |
| `encoding` | `hex` (lowercase) or `base64`, for the digest that goes on the request. A body or header hash is always lowercase hex. |
| `headers` | Headers set on the request BEFORE it is signed, each a template over the same facts less `signature.headers_hash`, which is computed after them and so is refused here. A scheme that signs a timestamp or a nonce carries it this way. |
| `signed_header_prefix` | The lowercase name prefix of the headers `signature.headers_hash` covers. Unset covers none, so the hash is of the empty string. |
| `place` | `basic` (`Basic base64(<key_id>:<digest>)`) or `digest` (the encoded digest on its own, after `prefix`). |
| `header` | The header the digest goes in; `authorization` unless named. |
| `prefix` | Text in front of the digest, under `place: digest` only. |

The facts a `canonical` or `headers` template may read, all of them the built
request's own:

| Fact | Value |
|------|-------|
| `signature.date` | The request's instant in RFC 2822, with a `-0000` zone. |
| `signature.timestamp`, `signature.timestamp_ms` | The same instant in Unix seconds and milliseconds. |
| `signature.nonce` | A fresh nonce per request, a UUIDv4 in hex. |
| `signature.method` | The method, uppercase. |
| `signature.host` | The host, lowercase, with the port when the URL states one. |
| `signature.path` | The path. |
| `signature.query` | The query pairs percent-encoded to the RFC 3986 unreserved set, then sorted and `&`-joined. |
| `signature.body_hash` | The body's digest as lowercase hex; the empty string's when there is no body. |
| `signature.headers_hash` | The digest of the signed headers -- each name lowercase, then its trimmed value, in name order, NUL-separated -- as lowercase hex. |
| `signature.key_id` | The instance's `key_id`, the public half naming the key. |

Duo's Admin API on the scheme Duo documents, which is what the shipped `duo`
profile carries:

```yaml
# profile
auth:
  accepts: [signature]
  signature:
    preset: duo_v5
```

Duo's legacy scheme, for a tenant whose endpoints still verify it. The
selection is on the INSTANCE, because a shipped profile is shared by every
instance of it and the version a tenant verifies is the tenant's:

```yaml
# instance
auth:
  mode: signature
  key_id: DIXXXXXXXXXXXXXXXXXX
  secret_key: "vault:<mount>/data/<path>:<key>"
  signature_preset: duo_v2
```

The typed `sources.duo` block spells the same choice `signature_version: v2`,
per connection, so one deployment can poll a tenant on each scheme.

Version 2 keys its HMAC with SHA-1, which is kept out of a security purpose
everywhere else in the platform, so selecting it is a deliberate exemption:
every connection that binds on it logs a warning at startup naming itself,
however the version was chosen. Duo's version 2 also signs a POST's BODY
parameters where this grammar signs the query, which nothing in the grammar can
say, so `duo_v2` paired with a unit that POSTs is refused at load rather than
signing every request over a string the provider will not recompute.

`duo_hmac` is still read as the mode's name and `integration_key` as `key_id`,
so an instance written against the earlier spelling binds unchanged. A PROFILE
on the old spelling had no `signature` block to write, so one that names the
mode and spells no scheme signs the `duo_v2` the name used to mean. Anything
that names a preset or spells a scheme out -- the shipped `duo` profile
included -- is unaffected.

Written out rather than named, `duo_v5` is:

```yaml
auth:
  accepts: [signature]
  signature:
    digest: sha512
    keying: hmac
    canonical:
      - "{{ signature.date }}"
      - "{{ signature.method }}"
      - "{{ signature.host }}"
      - "{{ signature.path }}"
      - "{{ signature.query }}"
      - "{{ signature.body_hash }}"
      - "{{ signature.headers_hash }}"
    headers:
      date: "{{ signature.date }}"
    signed_header_prefix: "x-duo-"
    place: basic
```

A scheme that hashes its key rather than keying an HMAC with it, signs a nonce
and a millisecond timestamp, and carries the digest bare -- Palo Alto Cortex
XDR's advanced key. One canonical line, because a template concatenates:

```yaml
auth:
  accepts: [signature]
  signature:
    digest: sha256
    keying: prefix
    canonical:
      - "{{ signature.nonce }}{{ signature.timestamp_ms }}"
    headers:
      x-xdr-nonce: "{{ signature.nonce }}"
      x-xdr-timestamp: "{{ signature.timestamp_ms }}"
      x-xdr-auth-id: "{{ signature.key_id }}"
    place: digest
```

Refused at load, each naming the field it sits on:

- a `preset` beside any other key of the block, because which half won would
  decide what goes on the wire
- no `canonical` and no `preset`
- a `canonical` or `headers` template that does not compile, or that reads any
  name but `signature` -- `vars`, `unit`, `window`, `page`, `key`, `item`,
  `auth`, `base_url`, `body`, `headers`, `ids`
- a `headers` key or a `header` that is not a header name
- a `prefix` under `place: basic`, whose value carries its own prefix, or one
  carrying what no header value may carry
- a `signed_header_prefix` that is not lowercase, because it is matched against
  lowercased names
- an instance with no `secret_key`, or with no `key_id` under `place: basic`,
  where the key id is the credential's user name
- a `signature_preset` on an instance whose mode is not `signature`

### What a signature may be about

A canonical string is COMPUTED per request, which is what makes signing its own
mode: a value composed once per instance cannot carry the request's own date,
method, path and body. What varies per request is therefore held to the
request: the `canonical` and `headers` templates read `signature.*` and the
grammar refuses them every other name, so no unit's `vars` or narrowing can
decide what another unit's request signs.

Nothing else about the mode varies at all. The digest, the keying, the
encoding, the placement, the header and the prefix are the profile's and are
not templates; the key id and the secret are the instance's and are read as
they were configured. So a per-unit value can reach neither the key nor where
the digest is written -- the two decisions that would otherwise let one unit
authenticate as another.

### What the token endpoint and the claims may read

A credential is minted once per instance, and once per scope a unit names, so
`token_url`, `gce_metadata.url`, `session_login.url` and its `body`, and every
`jwt_bearer` or `client_assertion` claim render against the INSTANCE's context
-- not per unit and not per request. They may read `vars`,
`base_url` and the `auth.*` the authenticator exposes. One that reads `unit`,
`window`, `page`, `key` or `item`, or that renders differently under a unit's
own `vars` or `base_url`, is refused when the instance binds: a claim deciding
the identity the token acts as (a domain-wide-delegation `sub`) must not be able
to vary by unit. `sigv4`'s `service` and `region` are the exception, and render
per request by design; `signature`'s templates render per request too, and are
held to the request's own facts instead ([What a signature may be
about](#what-a-signature-may-be-about)).

### Endpoint rules the modes are held to

- `early_refresh_secs` is capped at half the token's lifetime. A margin at or
  over the lifetime is logged; one between half and full is capped silently.
- `token_url` must be an https URL or a loopback address, because the client
  secret is a field of the form that is posted. A self-hosted deployment reached
  over plain http stops fetching, which is deliberate. `session_login.url` is
  held to the same rule, for the same reason: it posts the password.
- `gce_metadata.url` is exempt from that rule -- every cloud serves its metadata
  over plaintext -- and is held to its host instead: a loopback or link-local
  address, or a documented metadata host name.

## Retry, errors and quota

| Field | Meaning |
|-------|---------|
| `retry.max_retries` | Attempts after the first. |
| `retry.min_backoff_ms`, `retry.max_backoff_ms` | First backoff and the ceiling. |
| `retry.retry_on` | Statuses that are retried: a code, or a class such as `5xx`. |
| `retry.never_retry` | Statuses never retried whatever `retry_on` says (every shipped profile lists 401 and 403). |
| `retry.retry_after_header` | Honour `Retry-After` on a retried response. |
| `retry.retry_non_idempotent` | Retry POST as well as GET; off by default, set where every POST of the profile is a read. |
| `retry.throttle_when` | What a refusal meaning "slow down" looks like where the status does not say so (`status` and `body_contains`, as AWS answers 400 with a `ThrottlingException` body): retried whatever `retry_on` lists, and counted `throttle` rather than a client error. A status in `never_retry` is never one. |
| `error.at` | JSON pointer to the error text in a non-2xx body; the whole body is used when unset. |
| `quota.headers` | Gauge-name suffix -> response header, surfaced as `dfe_fetcher_api_quota_<suffix>` labelled by source. |

A final retry failure fails the tick, and a failed tick does not advance the
window. A status listed in an endpoint's `ignore_status` is neither retried nor
counted as an API error.

## Window

| Field | Meaning |
|-------|---------|
| `window.format` | How `window.start` and `window.end` render: `rfc3339_secs`, `rfc3339_millis`, `epoch_secs`, `epoch_millis` or `strftime:<pattern>`. |
| `window.step` | Split the window into steps of at most this length (`<n>s\|m\|h\|d`), one page sequence each. |
| `window.lookback` | Window used when the scheduler passes none. |
| `window.delay` | How long after the fact the provider publishes a record (`<n>s\|m\|h\|d`). Every window ends at least this long before now, so the cursor never passes a record not yet visible. Duo's is `2m`. |
| `endpoints[].window.format` | A unit's own rendering when its API differs from the profile's. |

A unit whose templates read no `window` runs once per tick, whatever the
step. A typed epoch number comes from the CEL cast: `"{{ int(window.start) }}"`
renders a JSON number.

## Endpoints

Each entry of `endpoints` is one unit of the source. `defaults` carries the
same keys (except `unit`, `shape`, `base_url`, `allow_hosts`, `auth`, `window`,
`vars`, `row_key`, `fail_when`, `add_fields`, `fold`, `lister`, `ignore_status`,
`timeout_secs`) and
every endpoint inherits them unless it sets its own; a `defaults.body` reaches
POST endpoints only, and an endpoint's own `prelude: []` or `construct: {}`
opts out of the defaults'.

| Field | Meaning |
|-------|---------|
| `unit` | Unit name: the `_source_fetcher` suffix, the dump `store` suffix, the topic suffix. |
| `shape` | Overrides the profile's shape for this unit. |
| `base_url` | This unit's base URL template when its host differs from the profile's. |
| `allow_hosts` | Origins this unit's rendered URLs may name on top of the profile's `allow_hosts`, which it widens rather than replaces. Held to the same rules. |
| `auth.scope` | The scope (OAuth2) or `scope` claim (JWT bearer) this unit's token is minted for. |
| `window.format` | This unit's window rendering. |
| `vars` | Values this unit's templates read as `vars.*`, over the instance's and under the instance's `units.<name>.vars`. |
| `method` | `GET` or `POST`; the defaults' when unset, else GET. |
| `path` | Path template appended to `base_url`; `{{ unit.name }}` lets one default path serve many units. |
| `query`, `headers` | Templates; an empty query rendering is omitted. |
| `body` | JSON body for POST; string leaves are templates, a leaf that is one expression keeps its type, a member that renders `null` is left out. |
| `rows` | Row framing ([Rows](#rows)). |
| `row_key` | JSON pointer to the row's identity, for the oversize stub and logs. |
| `paginate` | Pagination ([Pagination](#pagination)). |
| `fail_when` | CEL over a 2xx body; true fails the tick with the text at `error.at`. |
| `max_pages` | Page ceiling per tick. A unit that reads the window, a dump and a lister fail when pages remain past it, so nothing is committed past rows never read (a windowed unit is read again in narrower halves). A unit that reads no window is cut short. |
| `rate` | The provider's rate limit for this unit's requests (`requests_per_sec`, a fraction for a limit slower than one a second), held to across its pages, window steps and ticks; the defaults' when unset, and unpaced when neither sets one. |
| `timeout_secs` | Total bound on one request of this endpoint, its body included. Refused on a unit whose decoder streams: a streamed store takes as long as it takes and the client's idle read timeout bounds each read instead. |
| `max_page_bytes` | Bytes a page-bounded decoder may buffer. |
| `add_fields` | Fields added to every row: a template, or a JSON object or array whose string leaves are templates; one that reads `key` renders per key of the unit's keyset. |
| `fold` | A folding row builder: the rows of one key become the one row it builds. |
| `lister` | A listing protocol standing in for the page fetch. |
| `construct` | How the unit's requests are multiplied ([Constructs and prelude](#constructs-and-prelude)). |
| `prelude` | Requests sent once per tick before the first page. |
| `ignore_status` | Non-2xx statuses answered as an empty page instead of a failure. |

## Rows

| Field | Meaning |
|-------|---------|
| `rows.decoder` | The framing: `json_array`, `ndjson`, `json_at`, `document`, `json`, `lines`, `csv`. `json_at`, `document`, `json` and `csv` read the whole page into memory, bounded by `max_page_bytes`; the others stream. |
| `rows.at` | JSON pointer to the row array (`json_at` only). A pointer that leads nowhere fails the unit with a decode error and the checkpoint does not move, so a 2xx error body or a changed response shape cannot read as an empty page. `null` there is an empty page. |
| `rows.when_absent` | What an `at` pointer that leads nowhere means (`json_at` only): `fail` (default) or `empty`, an empty page. Set `empty` only where the provider documents omitting the key when there is nothing to list: shipped for `google_workspace`, `gcp` (`entries`, `scc`) and `gcp_pubsub`. |
| `rows.gzip` | The body is a gzip member sequence to inflate before framing (a `Content-Encoding: gzip` response is inflated by the client regardless). |
| `rows.header` | The first CSV record names the columns. |
| `rows.quoted` | Each framed row is a JSON string whose content is the row; it is unquoted first. |
| `rows.builder` | A row builder applied to each framed row: `columnar_table`, `cloudwatch_metrics`, `wrap_non_object`, `pubsub_message`, `cloudtrail_event`, `event_time`. The folding builder `go_module_aggregate` is named under `fold`, never here. |

## Pagination

| Field | Meaning |
|-------|---------|
| `paginate.strategy` | `none`, `link_header`, `cursor`, `page_number`, `offset`, `request_path`. |
| `paginate.from` | Where the token or URL comes from: `body:/pointer` or `header:Name`. |
| `paginate.into` | Where the token goes on the next request: `query:name`, `body:/pointer`, `body_replace:/pointer` (the next body is the token and nothing else) or `path`. |
| `paginate.stop_when` | CEL over `body` and `headers`; true stops after this page. |
| `paginate.total_pages_at` | JSON pointer to the total page count (`page_number`). |
| `paginate.total_at` | JSON pointer to the total row count (`offset`). |
| `paginate.page_size` | Rows per page (`offset`); a shorter page is the last; optional when `total_at` is set. |
| `paginate.param` | Query parameter carrying the page number or offset. |
| `paginate.start` | First page number or first offset. |
| `paginate.base` | Base URL a relative `request_path` value is resolved against; without one it resolves against the page's own URL. |

A cursor that is a list of scalars is sent comma-joined; an empty list, null
or a list of objects ends the sequence.

## Constructs and prelude

A unit takes at most one of `lookup`, `manifest` and `queue`; `keyset` can sit
beside any of them. A secondary request (a keyset request, a lookup batch, a
manifest item, a prelude step) is a `LookupRequest`: `method` (unset takes the
construct's default: POST for a lookup batch, a keyset request and a prelude
step, GET for a manifest item), `path` (appended to `base_url`, or a whole URL
when it renders one, whose host is held to the rule under
[Templates](#templates)), `query`, `headers`, `body` and `ignore_status`.

| Construct | Fields | Context |
|-----------|--------|---------|
| `construct.keyset` | `from` (a template rendering the list of keys) or `request` plus `keys_at` (the keys come from one request the unit sends first); exactly one. One page sequence per key. | `{{ key }}` in the unit's templates and `add_fields`. |
| `construct.lookup` | `id_at` (pointer into each first-stage row to its id; unset when the row is the id), `batch`, `request`, `rows`, `paginate`, `max_pages`. A second request per batch of ids; a lookup whose request reads `key` flushes its ids before the next key. | `{{ ids }}` in the batch request. |
| `construct.manifest` | `item_request`, `rows`, `key` and `position` (templates of the item's identity and RFC 3339 position, set together, giving every row an item checkpoint mark), `max_items` (items per key per tick: with `key` and `position` the rest wait for the next tick, otherwise a unit that reads the window or is a dump fails like a page ceiling, and one that reads no window is cut short), `add_fields` (rendered per item). The pages' rows are pointers; the unit's rows are what each item answers. | `{{ item }}` in the item request and fields. |
| `construct.queue` | `ack_at` (pointer into each framed row to its ack id), `ack_request` (POST by default), `ack_batch` (ids per acknowledgement). The ack request is sent for each batch of ids the driver hands back after the batch is delivered. | `{{ ids }}` in the ack request. |
| `prelude` | A list of `LookupRequest`s sent once per tick before the unit's first page, rendered from the unit's vars alone (never the window, a key or an item). | -- |

## Probe

`probe` names the API's cheapest authenticated call: `method`, `path`,
`query` and an optional `fail_when` (CEL over the 2xx body, for an API that
answers a bad credential inside a 200). Without a probe the health check
resolves the credential and, for a minting mode, exchanges it for a token.

## Templates

Templates are `{{ cel }}` expressions over:

| Name | Available |
|------|-----------|
| `vars.*` | The profile's defaults, overlaid by the instance's `vars`, the endpoint's `vars`, then the instance's `units.<name>.vars`. |
| `base_url` | The rendered base URL; the instance's in a `token_url` template, the unit's everywhere else. |
| `window.start`, `window.end` | The step's bounds, rendered per `window.format`; `int(...)` gives a number. |
| `unit.name` | The unit's name. |
| `key` | The current keyset key. |
| `item` | The current manifest item. |
| `ids` | The current lookup or ack batch, a typed list. |
| `auth.*` | What the mode exposes: `client_email`, `token_uri`, `token_url` for a JWT bearer, plus the token-response fields listed under `expose`. |

A body leaf that is exactly one expression keeps its JSON type; a mixed leaf
is a string.

A rendered path that is a whole URL replaces `base_url`, and its host is checked
before the request is signed. It must be the host of the unit's own `base_url`,
of the instance's, of an `auth.*` field the mode exposes (a token exchange
naming the org's own host, as Salesforce does), or of an `allow_hosts` entry.
Any other host fails the tick with a source error naming that host, so a value
the provider chose -- a manifest item's field, a key read out of a response --
cannot send the credential somewhere else, in the clear or at all. The same
check covers the URL a `link_header`, a `cursor` with `into: path` or a
`request_path` pager hands back.

## Instance

An instance (`sources.rest.<id>`, or a typed block mapped at load) carries:

| Field | Meaning |
|-------|---------|
| `enabled` | Whether the instance runs. |
| `profile` | A shipped profile by name, or the profile itself inline. |
| `interval_secs` | Fetch interval; the scheduler default when unset. |
| `topic` | Topic base; dump units land on `<topic>-<unit>`. |
| `filter` | CEL keep-filter over each row, hot-reloaded. |
| `auth` | The mode and its credentials (below). |
| `vars` | Values the profile's templates read as `vars.*`. |
| `units` | Per-unit narrowing, keyed by unit name (below). |
| `accumulate` | Batch bounds for this instance; the deployment's when unset. |

Every secret in `auth` is a credential spec (`vault:<mount>/data/<path>:<key>`,
`env:VAR`, or a literal) resolved when first used:

| Mode | Instance fields |
|------|-----------------|
| `bearer` | `token` |
| `api_key` | `key` |
| `basic` | `username`, `password` |
| `oauth2_client_credentials` | `client_id` (the literal id), then exactly one of `client_secret` or `private_key` (a bare RSA PEM signing a client assertion, where the profile declares one), optional `private_key_id` (the `kid` naming that key, not a secret) and optional `scope` overriding the profile's |
| `signature` | `key_id` (the public half naming the key -- Duo's integration key; also accepted as `integration_key`), `secret_key`, optional `signature_preset` in place of the profile's scheme ([Signing a request](#signing-a-request)) |
| `jwt_bearer` | exactly one of `service_account_key` (a Google-style key JSON as a spec), `service_account_key_file` (a spec resolving to the path of such a file), `private_key` (a bare RSA PEM for an API whose issuer and audience come from `vars`) |
| `gce_metadata`, `none` | nothing |
| `sigv4` | `access_key_id` and `secret_access_key`, or `credentials_json` alone (a document carrying both, in either the snake_case or the AWS `AccessKeyId` / `SecretAccessKey` spelling) |
| `credentials` | `credentials`, a spec per name the profile's placements read ([More than one credential on a request](#more-than-one-credential-on-a-request)) |
| `session_login` | `username`, `password` ([Logging in for a session token](#logging-in-for-a-session-token)) |

`vault:`, `bao:`, `openbao:`, `env:` and `file:` resolve, and the prefix is
matched exactly: an `aws:` spec, which needs a secrets feature the fetcher does
not build, or a near miss such as `Vault:`, is refused at load naming the field
and the prefix rather than reaching the provider as literal text.

`units.<name>` narrows one unit: `enabled`, `endpoint` (instantiate a
profile endpoint under this name with these overrides, so one endpoint serves
many buckets or subscriptions, tagged `<connection>.<name>`), `topic` (in
place of the instance's), `query` and `headers` (merged over the endpoint's),
`vars` (over the instance's for this unit).

## Validation

Binding an instance to its profile reports every problem with its field path
rather than the first: a mode the profile does not accept, a missing
credential field for the mode, a `units.<name>` the profile has no unit for
-- the refusal names the units the profile declares -- or one that
instantiates an endpoint under a name the profile already declares,
an empty `topic`, a filter or template that does not compile, a `base_url`
that is a literal, an `allow_hosts` entry that is a literal, a wildcard, or
renders something that is not an absolute URL, a pointer that does not start
with `/`, an `offset` pager
without `page_size` or `total_at`, a keyset with both or neither of `from` and
`request`, `key` in `add_fields` without a keyset, a queue unit that also
declares a lookup or manifest, a `defaults.body` on a GET, an `expose`
entry naming a credential, and a malformed or unmatched credential placement
([More than one credential on a request](#more-than-one-credential-on-a-request)).