The REST profile grammar

A profile is the shape of an API and carries no identity: a base URL template, the auth modes the API accepts, headers, the retry policy, where errors live in a non-2xx body, the window format, and the endpoints with their decoder and pager. An instance is one deployment of that shape: its credential kind and refs, vars, topic and interval, with per-unit narrowing. Every struct rejects unknown keys and every enum is a closed vocabulary, so a typo or an unsupported mode is a load error carrying the YAML line, never a setting that parses and does nothing.

The grammar as built is crates/rest/src/profile/mod.rs, and the generated JSON Schema of the whole config surface (this grammar included) is docs/config-schema.yaml. The shipped profiles under crates/fetcher/profiles/ are the worked examples of every entry below; config.example.yaml shows an instance of one and an inline profile. Defaults are not repeated here: read them off the Default impls in the grammar.

• Vocabulary
• Profile
• Auth
• Retry, errors and quota
• Window
• Endpoints
• Rows
• Pagination
• Constructs and prelude
• Probe
• Templates
• Instance
• Validation

Vocabulary

Every closed enum of the grammar, with the shipped profiles that use each entry.

Axis	Vocabulary	Shipped profiles that use it
Auth mode (auth.accepts, instance auth.mode)	none	pypi, crates_io, go_modules
	bearer (static token)	github, cloudflare, onepassword, slack, okta, runzero, gcp
	api_key (named header with a prefix, or a query parameter)	okta (SSWS)
	basic	none shipped
	oauth2_client_credentials (cached until shortly before expiry; per-unit auth.scope; expose allow-lists token-response fields for templates; client_assertion authenticates the exchange with a private-key JWT in place of a client secret)	azure, m365, bitwarden, crowdstrike, salesforce, runzero, okta
	jwt_bearer (RFC 7523 RS256 assertion from a service-account key, a key file or a bare PEM; claim templates)	gcp, gcp_pubsub, google_workspace, salesforce
	gce_metadata (the workload's token from the metadata server)	gcp, gcp_pubsub
	signature (a keyed digest over a canonical string of the request, in the digest, canonical shape and placement the profile names; Duo's versions are presets of it)	duo
	sigv4 (AWS Signature v4, service and region rendered per request)	aws, object_store
	credentials (more than one credential on the one request, each where the profile places it; a placement may compose several into one value)	datadog
	session_login (a login call mints a session token, read from a JSON pointer or a response header and carried in a header of the appliance's naming)	none shipped
Pager (paginate.strategy)	none	runzero, azure log_analytics
	link_header (RFC 5988 rel="next")	github, okta
	cursor (a token from the body or a header, injected into query:, body:, body_replace: or the path, and a list cursor is comma-joined)	aws, bitwarden, crowdstrike, datadog, duo, gcp, google_workspace, onepassword, slack
	page_number (bounded by total_pages_at)	cloudflare
	offset (advanced by rows received, bounded by total_at or a short page)	crowdstrike
	request_path (the next URL from the body or a header, relative resolved against the page)	azure, m365, salesforce
Decoder (rows.decoder)	json_array (streamed)	github, okta, m365
	ndjson (streamed)	runzero, object_store
	json_at (a wrapped page, rows at a JSON pointer)	aws, azure, bitwarden, cloudflare, crowdstrike, datadog, duo, gcp, gcp_pubsub, google_workspace, onepassword, salesforce, slack
	document (the whole body is one row)	pypi, crates_io, go_modules, aws cloudwatch_metrics
	json (an array is its elements, anything else one row)	object_store
	lines (plain text, one row per line)	go_modules, object_store
	csv (one object per record, optional header)	salesforce event_log_file
Row flags (rows.*)	gzip (inflate a gzip member sequence before framing), quoted (each row is a JSON string holding the row), header (CSV)	object_store, aws config, salesforce
Construct (construct.*, at most one of the last three per unit)	keyset (one page sequence per key of a list, from a template or a first request)	aws, azure, pypi, crates_io, go_modules
	lookup (a second request per batch of ids the pages yield, with its own rows and pager)	aws guardduty, aws cloudwatch_metrics, crowdstrike
	manifest (a second request per item the pages point at; key and position give each row an item checkpoint)	m365, object_store, salesforce, go_modules
	queue (rows carry an ack id; the ack request goes out after delivery)	gcp_pubsub
prelude	idempotent requests sent once per tick before the first page	m365 (the subscription start)
Row builder (rows.builder, or fold for a folding one)	columnar_table	azure log_analytics
	cloudwatch_metrics	aws cloudwatch_metrics
	go_module_aggregate (a fold)	go_modules
	wrap_non_object	object_store
	pubsub_message	gcp_pubsub
	cloudtrail_event	aws cloudtrail
	event_time	azure entra_signins, entra_directory_audits, entra_provisioning
Lister (lister)	s3 (ListObjectsV2, the keys modified after the unit's checkpoint)	object_store
Window (window.format)	rfc3339_secs, rfc3339_millis, epoch_secs, epoch_millis, strftime:<pattern>; step chunks the window, lookback is the window when the scheduler passes none; a unit may override the format	every incremental profile
Method	GET (retried by default), POST (retried only with retry.retry_non_idempotent)	every profile
Shape (shape, per profile or per endpoint)	incremental (rows are events in the window), dump (rows are the whole store, enveloped)	runzero is the shipped dump
Probe	the cheapest authenticated call, with an optional fail_when; without one the health check resolves the credential (and mints a token)	github, okta, cloudflare, duo, slack, onepassword, aws, pypi, crates_io, go_modules

Profile

Field	Meaning
profile	Registry key of a shipped profile; an inline profile may leave it empty.
maturity	alpha, beta or stable; the fetcher warns at startup for an enabled non-stable source.
base_url	Template; must reference a variable, never a literal URL, because a profile carries no identity. Rendered per unit, so a unit's vars can pick the host.
allow_hosts	Origins a rendered URL may name besides base_url's, for an API that answers on a second host (a sovereign cloud repointing management_url). Each entry is a template like base_url, so the host comes from the instance's vars, and names one exact origin: there are no wildcards, because a suffix match lets example.com.evil.net pass for example.com. Empty by default, and no shipped profile declares one.
shape	Default shape of every endpoint (incremental or dump).
auth	Accepted modes and the shape of each (Auth).
headers	Header templates on every request.
retry, error, quota	Retry, errors and quota.
window	Window.
probe	The health-check request (Probe).
defaults	Values every endpoint inherits unless it sets its own (Endpoints).
endpoints	The units.
vars	Default vars an instance may override.

Auth

auth.accepts lists the modes an instance may pick; the sibling keys carry the shape of each mode. Identity (tokens, keys, secrets) lives on the instance, never here.

Mode shape	Fields
api_key	header (the header carrying the key), query (the query parameter carrying it), prefix (text put in front of the key in a header, e.g. "SSWS "; a query parameter carries the key on its own, so a prefix beside query is refused rather than dropped).
oauth2_client_credentials	token_url (template), scope (omitted from the form when empty), expires_in_fallback_secs (lifetime assumed when the response carries no expires_in), early_refresh_secs, expose (top-level token-response fields the templates read as auth.<name>; access_token, refresh_token, id_token and client_secret are refused), client_assertion (see Authenticating the exchange with a key instead of a secret). The exchange posts grant_type=client_credentials, client_id, client_secret and the scope as a form.
jwt_bearer	token_url (template; may read auth.token_uri from a service-account key), claims (templates for iss, scope, aud and an optional sub; a claim that renders empty is left out), ttl_secs (exp - iat), expires_in_fallback_secs, early_refresh_secs, expose. The authenticator exposes client_email and token_uri from a service-account key and token_url once rendered.
gce_metadata	url (the service account's token URL on the metadata server, a template), expires_in_fallback_secs, early_refresh_secs.
sigv4	service and region (both templates, rendered per request from the unit's context; the body's SHA-256 is the payload hash).
signature	preset (a shipped scheme by name), or the scheme spelled out: digest, keying, canonical, encoding, headers, signed_header_prefix, place, header, prefix. See Signing a request.
credentials	A list of placements: header or query (exactly one) with an optional prefix, carrying either from (one named credential as it resolved) or value (a value composed of {{ credentials.<name> }} placeholders). See More than one credential on a request.
session_login	url (the login endpoint, a template), send, username_field / password_field, body, token_at or token_header, ttl_secs, early_refresh_secs, header, prefix. See Logging in for a session token.
bearer, basic, none	No profile-side shape.

A mode that mints a token for a scope (oauth2_client_credentials, jwt_bearer) lets a unit ask for its own with endpoints[].auth.scope; units with the same scope share one token.

Authenticating the exchange with a key instead of a secret

A client secret is a static credential the deployment holds and the exchange posts. Where a provider offers it, a key pair is the better trade: the private half never leaves the deployment, and the grant it buys is scoped to the integration rather than carrying the permissions of whoever minted a token. Okta steers its API Services apps this way and calls it private_key_jwt; Microsoft Entra, Zscaler OneAPI and Snowflake all take the same shape.

It is not a mode of its own. The exchange is still client credentials, so the profile declares client_assertion inside the oauth2_client_credentials block and the exchange posts grant_type=client_credentials, client_assertion_type naming RFC 7523's JWT-bearer assertion, the signed client_assertion and the scope, in place of the client_secret:

# profile
auth:
  accepts: [api_key, bearer, oauth2_client_credentials]
  oauth2_client_credentials:
    token_url: "{{ base_url }}/oauth2/v1/token"
    scope: okta.logs.read
    client_assertion:
      ttl_secs: 300

# instance
auth:
  mode: oauth2_client_credentials
  client_id: "0oa1example"
  private_key: "vault:<mount>/data/<path>:<key>"
  private_key_id: "the kid the app holds the public half under"

Field	Meaning
ttl_secs	Assertion lifetime, exp - iat. An hour at most -- a longer one is refused at load, because Okta answers invalid_client rather than shortening it. Short is right anyway: an assertion is spent the moment it is posted.
claims	Claim templates added to the assertion, or replacing iss, sub or aud. Each reads vars, base_url and auth.token_url / auth.client_id, and is held to the instance's context like every other credential template. The expose fields of the token response are NOT among them: they arrive with the token the assertion is minting.

The assertion always carries iss and sub as the client id and aud as the rendered token endpoint, which is what RFC 7523 s2.2 requires of a client authenticating as itself; iat and exp come from ttl_secs, and jti is fresh on every mint, because Okta refuses an assertion whose id it has already seen. A profile that writes iat, exp or jti in claims is refused at load rather than having them overwritten.

The signature is RS256, so private_key is an RSA private key PEM and a key of another kind is refused when it is read, naming the kind that is wanted. The instance's private_key_id becomes the kid of the JOSE header. An app holding ONE registered key pair resolves an assertion without it; an app holding two -- which is what rotating a signing key looks like, the new public key registered beside the old -- resolves by kid alone and refuses an assertion that names none. So an instance whose app has ever been rotated carries it.

Declaring the block says the API accepts the path. Which path an instance takes is decided by the credential it carries -- client_secret posts the secret, private_key signs the assertion -- so one deployment of an API that takes both may hold a secret and another a key pair. An instance carrying neither or both is refused when it binds, as is a private_key against a profile that declares no client_assertion.

More than one credential on a request

Several prominent APIs authenticate a call with two credentials, and there is no config workaround: a static added header is not a secret reference, and a key does not go in plain config. The credentials mode is the shape for them. The PROFILE declares where each credential goes and the name it is read under; the INSTANCE supplies a spec per name, because a profile carries no identity. Each name resolves once however many placements carry it, and every placement is applied to the one built request.

auth.credentials is the mode for credentials the deployment supplies as specs and the profile places as they resolved. Where a credential goes is an axis of a request rather than a mode of its own, so a mode that MINTS carries its own placement under its own key: session_login names the header and prefix its token goes in, over a token it minted rather than over a spec.

Two credentials in two headers of their own -- Datadog, where the api key names the organisation and the application key scopes the call to a user, so neither can be dropped:

# profile
auth:
  accepts: [credentials]
  credentials:
    - { header: DD-API-KEY, from: api_key }
    - { header: DD-APPLICATION-KEY, from: application_key }

# instance
auth:
  mode: credentials
  credentials:
    api_key: "vault:<mount>/data/<path>:<key>"
    application_key: "vault:<mount>/data/<path>:<key>"

Two credentials composed into ONE header value -- Tenable Vulnerability Management, whose Authorization carries both halves of the key pair, so the single-header shape does not help. Tenable Security Center is the same shape under x-apikey with the field names lower-cased:

# profile
auth:
  accepts: [credentials]
  credentials:
    - header: Authorization
      value: "accessKey={{ credentials.access_key }};secretKey={{ credentials.secret_key }}"

# instance
auth:
  mode: credentials
  credentials:
    access_key: "vault:<mount>/data/<path>:<key>"
    secret_key: "vault:<mount>/data/<path>:<key>"

One credential goes in as from with a prefix when the API wants text in front of it ({ header: Authorization, from: token, prefix: "Token " }). A value naming one credential says the same thing the long way round, so keep value for the several a provider wants in the one place.

A name is letters, digits, _ and - -- in the profile's from, in a {{ credentials.<name> }} placeholder and as an instance key alike, because the two sides are matched against each other. A spec is any the resolver reads; the spellings are in config.example.yaml.

A value is composed, not computed. Its only substitution is {{ credentials.<name> }}: anything else between the braces is refused at load, so no expression ever runs over a secret and no context -- vars, unit, window -- can reach a credential value. That is also why a composed value needs no per-unit check of its own: there is nothing in it a unit could change. The composed string is built once, on first use, and the placement that writes it marks the header sensitive, as it does a single credential.

Refused at load, each naming the field it sits on -- a placement by its index:

• a placement that names neither or both of header and query, or neither or both of from and value
• a header that is not a header name, an empty query or from, or a from that is not a credential name
• two placements writing the one header, where the second would replace the first, or the one query parameter, where both would be sent
• a query the profile writes itself -- defaults.query, an endpoint's query, the pager's into: query:<name> or its param -- or that a unit's narrowing writes: a query pair is appended, so the credential would be sent twice
• a value in a query: a composed value carries more than one secret and a query parameter is written into every access log and proxy on the path, so it goes in a header. A single credential may still go in a query.
• a prefix beside a value, which says the whole header value already, or beside a query, which carries the credential on its own
• a prefix or a value's literal text carrying what no header value may carry -- a newline, a tab -- checked here because the text is the profile's and static, so it is named with its field rather than failing on every tick
• a name the profile places and the instance supplies no spec for, and a name the instance supplies that no placement reads -- so a misspelling is refused from both ends rather than authenticating as half a credential

Logging in for a session token

An appliance rarely has an OAuth2 endpoint. It has a login call: post the administrator's credentials, get a token back, send that token on every later request. vCenter, Cisco Catalyst Centre, Check Point Management, F5 BIG-IP, Wazuh, Veeam, Zscaler's legacy API, Redfish and iDRAC BMCs and Qualys all work this way, and none of them speaks RFC 6749: the token comes back under a name the vendor chose, in the body or in a header, usually with nothing said about how long it lives.

So session_login reads the token from where the profile points, holds it for the lifetime the profile declares, and carries it in the header the profile names. It is NOT a bearer, and the grammar does not pretend otherwise.

Field	Meaning
url	The login endpoint, a template (may read base_url).
send	body puts the credentials in the JSON body, basic in the login's own Authorization: Basic header.
username_field, password_field	The body members the credentials are written into under send: body; username and password unless the appliance names them otherwise.
body	Further members of the login body, each a template (F5's loginProviderName). The profile's own text -- a credential never travels here. Refused under send: basic, which posts no body at all.
token_at	JSON pointer to the token in the login response. The empty pointer is the whole body, which is what vCenter answers: a JSON string that IS the session id.
token_header	The response header carrying the token instead (F5's X-F5-Auth-Token). Exactly one of this and token_at.
ttl_secs	How long the token is held. The appliance does not say, so this is the session lifetime it documents.
early_refresh_secs	How long before that lifetime is up the login runs again, with the same meaning it has on the OAuth2 spec: capped at half the lifetime.
header	The request header every later request carries the token in. Required -- there is no default, because there is no convention.
prefix	Text in front of the token in that header, for the appliances that want one.

The instance supplies username and password, the same two fields the basic mode takes.

# profile
auth:
  accepts: [session_login]
  session_login:
    url: "{{ base_url }}/mgmt/shared/authn/login"
    send: body
    body: { loginProviderName: tmos }
    token_at: /token/token
    ttl_secs: 1200
    header: X-F5-Auth-Token

# instance
auth:
  mode: session_login
  username: "svc-dfe"
  password: "vault:<mount>/data/<path>:<key>"

One login per instance: url and every body member render against the INSTANCE's context, and one that reads a name only a unit supplies is refused when the instance binds, as a token endpoint's is. A refused login carries its status and nothing of the response text -- a login body holds the password, and an appliance that answers a bad request by quoting what it was posted would put that password in the one line every consumer logs.

url is held to the same rule as a token endpoint: https, or plain http on a loopback address so a fixture needs no certificate. Plain http anywhere else is refused however private the network, and so is any other scheme. An appliance on a management LAN is exactly where a captured administrator password is worth the most, and every appliance here serves https -- with its own certificate, which is a trust-store question and not a reason to send the password in the clear. The refusal is at BIND rather than at the first login, because the URL is rendered there.

A login the appliance REFUSES is held for a few minutes and answered from rather than posted again. vCenter, F5 and Check Point all lock an account after a handful of failed logins, so a wrong or rotated password must not post the administrator's credentials once per unit per tick for as long as the deployment runs. An appliance that could not be reached is tried again by the next request, because that is not a failure the account is locked for.

Signing a request

Some providers authenticate a call with a digest of the call itself rather than with a credential the request carries. The digest algorithm, the canonical string it is taken over and where the digest goes are per-vendor details, so they are the signature mode's config rather than a mode of their own.

Field	Meaning
preset	A shipped scheme by name: duo_v5 or duo_v2. A preset is the WHOLE scheme, so a block that names one and also sets another key is refused.
digest	sha1, sha256 or sha512. The same digest takes the body hash and the header hash. sha1 is there for Duo's legacy scheme and is never a default.
keying	hmac (the secret keys an HMAC over the canonical string, RFC 2104) or prefix (the secret is hashed in front of the canonical string, with no HMAC).
canonical	The canonical string, ONE LINE per entry, joined by newlines. Each is a template over signature.* and nothing else.
encoding	hex (lowercase) or base64, for the digest that goes on the request. A body or header hash is always lowercase hex.
headers	Headers set on the request BEFORE it is signed, each a template over the same facts less signature.headers_hash, which is computed after them and so is refused here. A scheme that signs a timestamp or a nonce carries it this way.
signed_header_prefix	The lowercase name prefix of the headers signature.headers_hash covers. Unset covers none, so the hash is of the empty string.
place	basic (Basic base64(<key_id>:<digest>)) or digest (the encoded digest on its own, after prefix).
header	The header the digest goes in; authorization unless named.
prefix	Text in front of the digest, under place: digest only.

The facts a canonical or headers template may read, all of them the built request's own:

Fact	Value
signature.date	The request's instant in RFC 2822, with a -0000 zone.
signature.timestamp, signature.timestamp_ms	The same instant in Unix seconds and milliseconds.
signature.nonce	A fresh nonce per request, a UUIDv4 in hex.
signature.method	The method, uppercase.
signature.host	The host, lowercase, with the port when the URL states one.
signature.path	The path.
signature.query	The query pairs percent-encoded to the RFC 3986 unreserved set, then sorted and &-joined.
signature.body_hash	The body's digest as lowercase hex; the empty string's when there is no body.
signature.headers_hash	The digest of the signed headers -- each name lowercase, then its trimmed value, in name order, NUL-separated -- as lowercase hex.
signature.key_id	The instance's key_id, the public half naming the key.

Duo's Admin API on the scheme Duo documents, which is what the shipped duo profile carries:

# profile
auth:
  accepts: [signature]
  signature:
    preset: duo_v5

Duo's legacy scheme, for a tenant whose endpoints still verify it. The selection is on the INSTANCE, because a shipped profile is shared by every instance of it and the version a tenant verifies is the tenant's:

# instance
auth:
  mode: signature
  key_id: DIXXXXXXXXXXXXXXXXXX
  secret_key: "vault:<mount>/data/<path>:<key>"
  signature_preset: duo_v2

The typed sources.duo block spells the same choice signature_version: v2, per connection, so one deployment can poll a tenant on each scheme.

Version 2 keys its HMAC with SHA-1, which is kept out of a security purpose everywhere else in the platform, so selecting it is a deliberate exemption: every connection that binds on it logs a warning at startup naming itself, however the version was chosen. Duo's version 2 also signs a POST's BODY parameters where this grammar signs the query, which nothing in the grammar can say, so duo_v2 paired with a unit that POSTs is refused at load rather than signing every request over a string the provider will not recompute.

duo_hmac is still read as the mode's name and integration_key as key_id, so an instance written against the earlier spelling binds unchanged. A PROFILE on the old spelling had no signature block to write, so one that names the mode and spells no scheme signs the duo_v2 the name used to mean. Anything that names a preset or spells a scheme out -- the shipped duo profile included -- is unaffected.

Written out rather than named, duo_v5 is:

auth:
  accepts: [signature]
  signature:
    digest: sha512
    keying: hmac
    canonical:
      - "{{ signature.date }}"
      - "{{ signature.method }}"
      - "{{ signature.host }}"
      - "{{ signature.path }}"
      - "{{ signature.query }}"
      - "{{ signature.body_hash }}"
      - "{{ signature.headers_hash }}"
    headers:
      date: "{{ signature.date }}"
    signed_header_prefix: "x-duo-"
    place: basic

A scheme that hashes its key rather than keying an HMAC with it, signs a nonce and a millisecond timestamp, and carries the digest bare -- Palo Alto Cortex XDR's advanced key. One canonical line, because a template concatenates:

auth:
  accepts: [signature]
  signature:
    digest: sha256
    keying: prefix
    canonical:
      - "{{ signature.nonce }}{{ signature.timestamp_ms }}"
    headers:
      x-xdr-nonce: "{{ signature.nonce }}"
      x-xdr-timestamp: "{{ signature.timestamp_ms }}"
      x-xdr-auth-id: "{{ signature.key_id }}"
    place: digest

Refused at load, each naming the field it sits on:

• a preset beside any other key of the block, because which half won would decide what goes on the wire
• no canonical and no preset
• a canonical or headers template that does not compile, or that reads any name but signature -- vars, unit, window, page, key, item, auth, base_url, body, headers, ids
• a headers key or a header that is not a header name
• a prefix under place: basic, whose value carries its own prefix, or one carrying what no header value may carry
• a signed_header_prefix that is not lowercase, because it is matched against lowercased names
• an instance with no secret_key, or with no key_id under place: basic, where the key id is the credential's user name
• a signature_preset on an instance whose mode is not signature

What a signature may be about

A canonical string is COMPUTED per request, which is what makes signing its own mode: a value composed once per instance cannot carry the request's own date, method, path and body. What varies per request is therefore held to the request: the canonical and headers templates read signature.* and the grammar refuses them every other name, so no unit's vars or narrowing can decide what another unit's request signs.

Nothing else about the mode varies at all. The digest, the keying, the encoding, the placement, the header and the prefix are the profile's and are not templates; the key id and the secret are the instance's and are read as they were configured. So a per-unit value can reach neither the key nor where the digest is written -- the two decisions that would otherwise let one unit authenticate as another.

What the token endpoint and the claims may read

A credential is minted once per instance, and once per scope a unit names, so token_url, gce_metadata.url, session_login.url and its body, and every jwt_bearer or client_assertion claim render against the INSTANCE's context -- not per unit and not per request. They may read vars, base_url and the auth.* the authenticator exposes. One that reads unit, window, page, key or item, or that renders differently under a unit's own vars or base_url, is refused when the instance binds: a claim deciding the identity the token acts as (a domain-wide-delegation sub) must not be able to vary by unit. sigv4's service and region are the exception, and render per request by design; signature's templates render per request too, and are held to the request's own facts instead (What a signature may be about).

Endpoint rules the modes are held to

• early_refresh_secs is capped at half the token's lifetime. A margin at or over the lifetime is logged; one between half and full is capped silently.
• token_url must be an https URL or a loopback address, because the client secret is a field of the form that is posted. A self-hosted deployment reached over plain http stops fetching, which is deliberate. session_login.url is held to the same rule, for the same reason: it posts the password.
• gce_metadata.url is exempt from that rule -- every cloud serves its metadata over plaintext -- and is held to its host instead: a loopback or link-local address, or a documented metadata host name.

Retry, errors and quota

Field	Meaning
retry.max_retries	Attempts after the first.
retry.min_backoff_ms, retry.max_backoff_ms	First backoff and the ceiling.
retry.retry_on	Statuses that are retried: a code, or a class such as 5xx.
retry.never_retry	Statuses never retried whatever retry_on says (every shipped profile lists 401 and 403).
retry.retry_after_header	Honour Retry-After on a retried response.
retry.retry_non_idempotent	Retry POST as well as GET; off by default, set where every POST of the profile is a read.
retry.throttle_when	What a refusal meaning "slow down" looks like where the status does not say so (status and body_contains, as AWS answers 400 with a ThrottlingException body): retried whatever retry_on lists, and counted throttle rather than a client error. A status in never_retry is never one.
error.at	JSON pointer to the error text in a non-2xx body; the whole body is used when unset.
quota.headers	Gauge-name suffix -> response header, surfaced as dfe_fetcher_api_quota_<suffix> labelled by source.

A final retry failure fails the tick, and a failed tick does not advance the window. A status listed in an endpoint's ignore_status is neither retried nor counted as an API error.

Window

Field	Meaning
window.format	How window.start and window.end render: rfc3339_secs, rfc3339_millis, epoch_secs, epoch_millis or strftime:<pattern>.
window.step	Split the window into steps of at most this length (<n>s|m|h|d), one page sequence each.
window.lookback	Window used when the scheduler passes none.
window.delay	How long after the fact the provider publishes a record (<n>s|m|h|d). Every window ends at least this long before now, so the cursor never passes a record not yet visible. Duo's is 2m.
endpoints[].window.format	A unit's own rendering when its API differs from the profile's.

A unit whose templates read no window runs once per tick, whatever the step. A typed epoch number comes from the CEL cast: "{{ int(window.start) }}" renders a JSON number.

Endpoints

Each entry of endpoints is one unit of the source. defaults carries the same keys (except unit, shape, base_url, allow_hosts, auth, window, vars, row_key, fail_when, add_fields, fold, lister, ignore_status, timeout_secs) and every endpoint inherits them unless it sets its own; a defaults.body reaches POST endpoints only, and an endpoint's own prelude: [] or construct: {} opts out of the defaults'.

Field	Meaning
unit	Unit name: the _source_fetcher suffix, the dump store suffix, the topic suffix.
shape	Overrides the profile's shape for this unit.
base_url	This unit's base URL template when its host differs from the profile's.
allow_hosts	Origins this unit's rendered URLs may name on top of the profile's allow_hosts, which it widens rather than replaces. Held to the same rules.
auth.scope	The scope (OAuth2) or scope claim (JWT bearer) this unit's token is minted for.
window.format	This unit's window rendering.
vars	Values this unit's templates read as vars.*, over the instance's and under the instance's units.<name>.vars.
method	GET or POST; the defaults' when unset, else GET.
path	Path template appended to base_url; {{ unit.name }} lets one default path serve many units.
query, headers	Templates; an empty query rendering is omitted.
body	JSON body for POST; string leaves are templates, a leaf that is one expression keeps its type, a member that renders null is left out.
rows	Row framing (Rows).
row_key	JSON pointer to the row's identity, for the oversize stub and logs.
paginate	Pagination (Pagination).
fail_when	CEL over a 2xx body; true fails the tick with the text at error.at.
max_pages	Page ceiling per tick. A unit that reads the window, a dump and a lister fail when pages remain past it, so nothing is committed past rows never read (a windowed unit is read again in narrower halves). A unit that reads no window is cut short.
rate	The provider's rate limit for this unit's requests (requests_per_sec, a fraction for a limit slower than one a second), held to across its pages, window steps and ticks; the defaults' when unset, and unpaced when neither sets one.
timeout_secs	Total bound on one request of this endpoint, its body included. Refused on a unit whose decoder streams: a streamed store takes as long as it takes and the client's idle read timeout bounds each read instead.
max_page_bytes	Bytes a page-bounded decoder may buffer.
add_fields	Fields added to every row: a template, or a JSON object or array whose string leaves are templates; one that reads key renders per key of the unit's keyset.
fold	A folding row builder: the rows of one key become the one row it builds.
lister	A listing protocol standing in for the page fetch.
construct	How the unit's requests are multiplied (Constructs and prelude).
prelude	Requests sent once per tick before the first page.
ignore_status	Non-2xx statuses answered as an empty page instead of a failure.

Rows

Field	Meaning
rows.decoder	The framing: json_array, ndjson, json_at, document, json, lines, csv. json_at, document, json and csv read the whole page into memory, bounded by max_page_bytes; the others stream.
rows.at	JSON pointer to the row array (json_at only). A pointer that leads nowhere fails the unit with a decode error and the checkpoint does not move, so a 2xx error body or a changed response shape cannot read as an empty page. null there is an empty page.
rows.when_absent	What an at pointer that leads nowhere means (json_at only): fail (default) or empty, an empty page. Set empty only where the provider documents omitting the key when there is nothing to list: shipped for google_workspace, gcp (entries, scc) and gcp_pubsub.
rows.gzip	The body is a gzip member sequence to inflate before framing (a Content-Encoding: gzip response is inflated by the client regardless).
rows.header	The first CSV record names the columns.
rows.quoted	Each framed row is a JSON string whose content is the row; it is unquoted first.
rows.builder	A row builder applied to each framed row: columnar_table, cloudwatch_metrics, wrap_non_object, pubsub_message, cloudtrail_event, event_time. The folding builder go_module_aggregate is named under fold, never here.

Pagination

Field	Meaning
paginate.strategy	none, link_header, cursor, page_number, offset, request_path.
paginate.from	Where the token or URL comes from: body:/pointer or header:Name.
paginate.into	Where the token goes on the next request: query:name, body:/pointer, body_replace:/pointer (the next body is the token and nothing else) or path.
paginate.stop_when	CEL over body and headers; true stops after this page.
paginate.total_pages_at	JSON pointer to the total page count (page_number).
paginate.total_at	JSON pointer to the total row count (offset).
paginate.page_size	Rows per page (offset); a shorter page is the last; optional when total_at is set.
paginate.param	Query parameter carrying the page number or offset.
paginate.start	First page number or first offset.
paginate.base	Base URL a relative request_path value is resolved against; without one it resolves against the page's own URL.

A cursor that is a list of scalars is sent comma-joined; an empty list, null or a list of objects ends the sequence.

Constructs and prelude

A unit takes at most one of lookup, manifest and queue; keyset can sit beside any of them. A secondary request (a keyset request, a lookup batch, a manifest item, a prelude step) is a LookupRequest: method (unset takes the construct's default: POST for a lookup batch, a keyset request and a prelude step, GET for a manifest item), path (appended to base_url, or a whole URL when it renders one, whose host is held to the rule under Templates), query, headers, body and ignore_status.

Construct	Fields	Context
construct.keyset	from (a template rendering the list of keys) or request plus keys_at (the keys come from one request the unit sends first); exactly one. One page sequence per key.	{{ key }} in the unit's templates and add_fields.
construct.lookup	id_at (pointer into each first-stage row to its id; unset when the row is the id), batch, request, rows, paginate, max_pages. A second request per batch of ids; a lookup whose request reads key flushes its ids before the next key.	{{ ids }} in the batch request.
construct.manifest	item_request, rows, key and position (templates of the item's identity and RFC 3339 position, set together, giving every row an item checkpoint mark), max_items (items per key per tick: with key and position the rest wait for the next tick, otherwise a unit that reads the window or is a dump fails like a page ceiling, and one that reads no window is cut short), add_fields (rendered per item). The pages' rows are pointers; the unit's rows are what each item answers.	{{ item }} in the item request and fields.
construct.queue	ack_at (pointer into each framed row to its ack id), ack_request (POST by default), ack_batch (ids per acknowledgement). The ack request is sent for each batch of ids the driver hands back after the batch is delivered.	{{ ids }} in the ack request.
prelude	A list of LookupRequests sent once per tick before the unit's first page, rendered from the unit's vars alone (never the window, a key or an item).	--

Probe

probe names the API's cheapest authenticated call: method, path, query and an optional fail_when (CEL over the 2xx body, for an API that answers a bad credential inside a 200). Without a probe the health check resolves the credential and, for a minting mode, exchanges it for a token.

Templates

Templates are {{ cel }} expressions over:

Name	Available
vars.*	The profile's defaults, overlaid by the instance's vars, the endpoint's vars, then the instance's units.<name>.vars.
base_url	The rendered base URL; the instance's in a token_url template, the unit's everywhere else.
window.start, window.end	The step's bounds, rendered per window.format; int(...) gives a number.
unit.name	The unit's name.
key	The current keyset key.
item	The current manifest item.
ids	The current lookup or ack batch, a typed list.
auth.*	What the mode exposes: client_email, token_uri, token_url for a JWT bearer, plus the token-response fields listed under expose.

A body leaf that is exactly one expression keeps its JSON type; a mixed leaf is a string.

A rendered path that is a whole URL replaces base_url, and its host is checked before the request is signed. It must be the host of the unit's own base_url, of the instance's, of an auth.* field the mode exposes (a token exchange naming the org's own host, as Salesforce does), or of an allow_hosts entry. Any other host fails the tick with a source error naming that host, so a value the provider chose -- a manifest item's field, a key read out of a response -- cannot send the credential somewhere else, in the clear or at all. The same check covers the URL a link_header, a cursor with into: path or a request_path pager hands back.

Instance

An instance (sources.rest.<id>, or a typed block mapped at load) carries:

Field	Meaning
enabled	Whether the instance runs.
profile	A shipped profile by name, or the profile itself inline.
interval_secs	Fetch interval; the scheduler default when unset.
topic	Topic base; dump units land on <topic>-<unit>.
filter	CEL keep-filter over each row, hot-reloaded.
auth	The mode and its credentials (below).
vars	Values the profile's templates read as vars.*.
units	Per-unit narrowing, keyed by unit name (below).
accumulate	Batch bounds for this instance; the deployment's when unset.

Every secret in auth is a credential spec (vault:<mount>/data/<path>:<key>, env:VAR, or a literal) resolved when first used:

Mode	Instance fields
bearer	token
api_key	key
basic	username, password
oauth2_client_credentials	client_id (the literal id), then exactly one of client_secret or private_key (a bare RSA PEM signing a client assertion, where the profile declares one), optional private_key_id (the kid naming that key, not a secret) and optional scope overriding the profile's
signature	key_id (the public half naming the key -- Duo's integration key; also accepted as integration_key), secret_key, optional signature_preset in place of the profile's scheme (Signing a request)
jwt_bearer	exactly one of service_account_key (a Google-style key JSON as a spec), service_account_key_file (a spec resolving to the path of such a file), private_key (a bare RSA PEM for an API whose issuer and audience come from vars)
gce_metadata, none	nothing
sigv4	access_key_id and secret_access_key, or credentials_json alone (a document carrying both, in either the snake_case or the AWS AccessKeyId / SecretAccessKey spelling)
credentials	credentials, a spec per name the profile's placements read (More than one credential on a request)
session_login	username, password (Logging in for a session token)

vault:, bao:, openbao:, env: and file: resolve, and the prefix is matched exactly: an aws: spec, which needs a secrets feature the fetcher does not build, or a near miss such as Vault:, is refused at load naming the field and the prefix rather than reaching the provider as literal text.

units.<name> narrows one unit: enabled, endpoint (instantiate a profile endpoint under this name with these overrides, so one endpoint serves many buckets or subscriptions, tagged <connection>.<name>), topic (in place of the instance's), query and headers (merged over the endpoint's), vars (over the instance's for this unit).

Validation

Binding an instance to its profile reports every problem with its field path rather than the first: a mode the profile does not accept, a missing credential field for the mode, a units.<name> the profile has no unit for -- the refusal names the units the profile declares -- or one that instantiates an endpoint under a name the profile already declares, an empty topic, a filter or template that does not compile, a base_url that is a literal, an allow_hosts entry that is a literal, a wildcard, or renders something that is not an absolute URL, a pointer that does not start with /, an offset pager without page_size or total_at, a keyset with both or neither of from and request, key in add_fields without a keyset, a queue unit that also declares a lookup or manifest, a defaults.body on a GET, an expose entry naming a credential, and a malformed or unmatched credential placement (More than one credential on a request).