# Hunt-runner scaling: deterministic-due, no engine dependency > The hunt runner is pull-based and multi-pod. **Default deployment is ONE steady pod** > (no ScaledObject at all) - the runner is a thin orchestrator (all hunt load > runs in ClickHouse), so one pod covers the large majority of deployments. > Autoscaling is **BETA and opt-in** (`huntRunner.autoscaling.enabled`). When > turned on, KEDA scales the pod count on the deterministic-due backlog this doc > explains, via the fail-safe `dfe-keda-shim` - never the engine, and no longer > a MySQL-on-ClickHouse scaler (retired, see "KEDA ScaledObject" below). **Keep autoscaling off (the default) on 2.2.2.** It reads the due backlog from the `hunt_schedule` table, and nothing in the 2.2.2 charts writes that table: there is no materialise Job, so the backlog always reads 0. With autoscaling on the runner never scales out, and at `minReplicaCount: 0` no hunt ever fires. To try it anyway, run `dfe-hunt-runner materialise` after every hunt-config change. This doc is the SCALING contract: how the due backlog is computed with zero ambiguity, how the beta ScaledObject wakes/grows/shrinks the pod count from it, and the property that matters most - the runner has NO operational dependency on dfe-engine. ## The one hard requirement: due-detection with zero workers This is the contract the BETA autoscaling relies on (see "Default deployment, beta autoscaling, and disable" below for when it applies - it also governs correctness if you opt into `minReplicaCount: 0`, i.e. scale-to-zero). KEDA can only wake the first worker if "is any hunt due right now?" is answerable with no worker running - i.e. as a query KEDA itself (via the shim) runs against ClickHouse. But the per-hunt schedule (interval + stable phase offset) otherwise lives only in a running worker's memory (loaded from the gitops hunt configs). At zero workers there is nobody to ask. So the schedule is MATERIALISED into a small ClickHouse table, `hunt_schedule`, at config-deploy time. The scaler then reads it directly. Three tables already exist for coordination (`ch_coordinator.py`): `hunt_lease`, `hunt_watermark`, `hunt_state`. `hunt_schedule` is the fourth, and the only one written outside the worker loop. ```text hunt_schedule(hunt_id, interval_seconds, phase_offset, enabled, updated) ReplacingMergeTree(updated) ORDER BY hunt_id ``` `enabled` is a soft tombstone: a deleted hunt is written `enabled=0` so it stops waking KEDA, without a DELETE. ## The scaler query (deterministic-due) KEDA runs `schedule.due_query(db)` against ClickHouse (via the fail-safe `dfe-keda-shim`'s `/keda/hunt-backlog` endpoint - see "KEDA ScaledObject" below) - the count of hunts that are due-and-unclaimed. It uses the EXACT arithmetic the worker uses (`schedule.latest_fire`): ```text boundary = intDiv(now(), interval) * interval -- start of this interval current = boundary + phase_offset -- this interval's scheduled fire fire = if(now() >= current, current, -- the latest fire at or before now current - interval) due = watermark < fire -- that fire not already completed AND lease_until <= now() -- not currently running ``` `due_count > 0` -> scale up, and `== 0` -> scale down (to zero only where `minReplicaCount: 0` has been opted in - the default floor is 1 pod). The `enabled=1` filter is applied in an inner subquery so a tombstone (interval=0) can never reach the `intDiv` (no divide-by-zero). ### Why the watermark comparison is exact The worker writes `watermark = scheduled_start` (the fire time) after a run commits (`worker.py`: `window(...)` returns `end = scheduled_start`, then `set_watermark(end)`). So the watermark is in the SAME epoch-fire scale as the boundary arithmetic - `watermark < fire` means precisely "this fire has not been completed". No unit mismatch, no data-time-vs-fire-time skew. ## Why it is RELIABLE Autoscaling (when the BETA is enabled) is only safe if KEDA's view of "due" never diverges from a worker's. It cannot, because both compute the same thing from the same inputs: 1. **Parity, tested.** The scaler predicate equals `runner.tick`'s own gate (via `schedule.latest_fire` + the watermark + lease checks) across a full matrix of (hunt_id, interval, now, watermark, lease), and the SQL implements that predicate against real ClickHouse - counting exactly the runnable hunts, excluding tombstoned / future-watermark / leased hunts, and returning 0 when every hunt is caught up. 2. **Single source of truth for the offset.** The materialiser computes `phase_offset` with the same `schedule.phase_offset` the worker uses, so the two can never drift. Change the offset algorithm once and both move together. 3. **Gate on the signal, not a timer.** KEDA polls the backlog query (via the shim - a real readiness signal), it does not race a clock. Worst-case wake latency is `pollingInterval + pod-start`. For hunts (minute-plus intervals) that is immaterial. This is the "gate on the real condition" rule from the testing standard applied to autoscaling. 4. **Self-healing.** If a worker claims a hunt then dies before writing the watermark, its lease expires (`lease_seconds`, default 300) and the fire is still owed (`watermark < fire`), so the scaler keeps >=1 worker and the hunt is reclaimed. Nothing double-runs (a hunt run twice writes and notifies twice): a claim stands only if acknowledged within a settle of its free read and still the latest row after it. A later claim is never released, as it may sit over the winner, but it holds nothing, so the next poll claims again. 5. **No thundering herd.** The phase offset spreads same-interval hunts across the interval, so `due_count` rises gradually and KEDA scales out smoothly rather than all-at-once at the boundary. The offset falls in the first `hunts.phase_spread_fraction` of the interval (default 0.8, env `DFE_ENGINE_HUNTS_PHASE_SPREAD_FRACTION`). The runner, the API's `next_due` and the materialised schedule all read it, so republish the schedule (`dfe-hunt-runner materialise`) after changing it. Bounded catch-up: a fire stays owed until its watermark is written, so a worker that ticks inside `[boundary, boundary+offset)` runs the previous interval's fire if nothing ran it, and the scaler counts it the same way. Without that, a hunt whose offset sits between two poll ticks (a 46s offset polled at :00/:15/:30/:45) never fires at all. Max lateness for any fire is under one interval - inherent to the phase-offset design, and identical whether the decision is made by KEDA or a resident worker. ## No operational dependency on dfe-engine This is the point of the pull-based design. Once deployed, the running system needs only ClickHouse: - **Workers** read hunt defs from the gitops config (mounted / cloned), coordinate through the four CH tables, and execute queries against CH. No engine call on the hot path. - **KEDA** reads `hunt_schedule` + the coordination tables from CH via the `dfe-keda-shim` - a separate, distinctly-labelled pod running the SAME engine container image under a different entrypoint (`dfe-keda-shim run`, not the live `dfe-engine` API service). No call to the running engine API. - **The engine's only role is at CONFIG-DEPLOY time**, not runtime: materialise `hunt_schedule` when the hunt config changes, with the one-shot, idempotent `dfe-hunt-runner materialise`. The 2.2.2 charts do not run it (see below). So if the engine pod is down, hunts keep firing, and so does KEDA scaling once the schedule is materialised. The only thing you lose while the engine is down is the ability to CHANGE configs - which needs the engine anyway. ClickHouse is DFE's single operational store, and the runner holds to that (no Postgres, no engine, no convenience store on the hot path), which is also what lets the same runner work on a non-k8s single deploy (dfe-docker) where only ClickHouse is guaranteed present. ### Coordination needs a SINGLE LOGICAL ClickHouse (scale-tier requirement) The four coordination tables are the shared source of truth every worker and KEDA read/write. That only works if all of them see the SAME tables. On the **scale tier's multi-node cluster behind a round-robin load balancer**, a plain `ReplacingMergeTree` (and a plain `CREATE DATABASE`) would land on one node, so workers whose connections land on other nodes would see an empty or absent table and coordination would silently break (`UNKNOWN_DATABASE`, missing leases). The schema phase therefore creates the coordination tables as `ReplicatedReplacingMergeTree` inside the Replicated `dfe` database on every deployment, so lease, watermark and schedule state is shared across nodes (). ### Materialisation (config-deploy time) `schedule.publish_schedule(ch, db, specs)` is the ONLY writer of `hunt_schedule`, and `dfe-hunt-runner materialise` is its only caller. It is idempotent and diff-aware (tombstones hunts that vanished from config). The 2.2.2 charts run no Job for it, so on Kubernetes the table stays empty unless an operator runs the command after each hunt-config change. Run that way, a brand-new hunt added while scaled to zero still gets a `hunt_schedule` row, so KEDA can wake a worker for it. ## KEDA ScaledObject (shape) **KEDA has no native ClickHouse scaler.** The earlier approach - KEDA's `mysql` scaler pointed at ClickHouse's MySQL-compatible interface (`mysql_port`, 9004) - has been **RETIRED**: KEDA's only failure knob for a DB scaler is a fixed `fallbackReplicas` count, not "hold at current", so a broken metric could scale the wrong way and blow cloud spend. It is replaced by a small OWNED, fail-safe adapter: the **`dfe-keda-shim`**, its own pod on the engine image (`dfe-keda-shim run`), labelled so the engine Service never routes to it. The published engine chart (`oci://ghcr.io/hyperi-io/charts/dfe-engine`) renders only the engine pod. The shim comes from `dfe-infra helm/charts/dfe-extras/templates/engine-keda-shim.yaml`, on the same values. KEDA's `metrics-api` scaler polls the shim over HTTP, and the shim runs `schedule.due_query(db)` against ClickHouse - the SAME SQL the worker gate uses (see "The scaler query" above, imported directly from `dfe_engine.hunts.schedule.due_query`, so it cannot drift) - and hands back one integer. ```mermaid flowchart LR subgraph CH["ClickHouse"] hs[("hunt_schedule +\nhunt_lease/watermark/state")] end shim["dfe-keda-shim\n(dfe-keda-shim run)\nGET /keda/hunt-backlog"] ko["KEDA metrics-api trigger\n(ScaledObject: dfe-hunt-runner-scaler)"] hr["dfe-hunt-runner Deployment\n(replicas: N)"] hs -->|"due_query(db)"| shim shim -->|"{value: due_count}"| ko ko -->|"AverageValue, target = targetBacklog"| hr shim -. "on failure: last-good\ncache / cold-hold" .-> ko ``` Ships with the hunt-runner Deployment, gated by `huntRunner.autoscaling.enabled` (default `false`, **BETA**). When enabled, the Deployment's `replicas` field is OMITTED (the ScaledObject owns the count instead) and a `dfe-hunt-runner-scaler` `ScaledObject` is rendered alongside it (`dfe-infra helm/charts/dfe-extras/templates/engine-hunt-runner.yaml`): ```yaml apiVersion: keda.sh/v1alpha1 kind: ScaledObject metadata: name: dfe-hunt-runner-scaler spec: scaleTargetRef: name: dfe-hunt-runner minReplicaCount: 1 # default 1, NOT 0 - set 0 via values to opt into scale-to-zero maxReplicaCount: 5 # default; huntRunner.autoscaling.maxReplicaCount cooldownPeriod: 300 pollingInterval: 30 triggers: - type: metrics-api metricType: AverageValue # ceil(due / targetBacklog) workers metadata: targetValue: "10" # huntRunner.autoscaling.targetBacklog # huntRunner.autoscaling.shimAddress, else the shim in the release's own namespace url: "http://dfe-keda-shim..svc.cluster.local:8080/keda/hunt-backlog" valueLocation: "value" ``` **The fail-safe** (`dfe_engine/keda_shim/shim.py`, `QueryShim.run`): every query caches its last-good integer per (query, params). On ANY failure - CH unreachable, timeout, a bad query - the shim returns the CACHED value, so KEDA sees no change and the pod count FREEZES at current, so it can never run replicas up on a broken metric. A cold start with no cache yet, plus a failure, falls back to a conservative `cold_hold` (0 for the backlog query - hold at `minReplicaCount`). This cache-or-cold-hold behaviour is exactly why the shim replaced the zero-code MySQL-on-ClickHouse route: KEDA's native DB scalers cannot express "hold at current", only a fixed fallback count. Other notes: the shim is a separate, distinctly-labelled pod (`app.kubernetes.io/name: dfe-keda-shim`) running the same engine container image under a different entrypoint - it is NOT the live engine API, so hunt-runner scaling still has no dependency on the engine service being up. Queries (including the backlog query's clamp/cold-hold/timeout) are config-driven (`keda_shim/queries.yaml` + an optional mounted override), so a schema change or a tuning change is a config edit, not a shim rebuild. ### Proving it: the artificial scale test `bootstrap/keda-scale-test.sh` (dfe-infra) is the mechanised proof that the shim -> KEDA scale-out/scale-in path actually works, rather than an aspirational claim: it forces the path on demand with a disposable target instead of waiting for real hunt load - stand up a throwaway Deployment + ScaledObject wired to the shim's pressure query for a test `ServiceName`, INSERT a high `dfe_scaling_pressure` row into `otel_metrics_gauge` (any `_scaling_pressure` spelling matches - see [keda-scaling](/v/2.2.2/dfe-engine/deployment/keda-scaling.md)), poll (bounded, 120s) for scale-OUT to >= 2 replicas, drag the rolling average back down, poll (bounded, 180s) for scale-IN to 1, then self-clean. It rides the SAME shim endpoint and metrics-api mechanism the hunt-runner's `/keda/hunt-backlog` trigger uses, so a pass proves the mechanism the hunt-runner beta relies on, not a mock. Wired into `bootstrap/run-all-smoke-tests.sh` (gated `DFE_KEDA_SCALE_TEST`, default on). ## Default deployment, beta autoscaling, and disable The hunt runner is a thin orchestrator - virtually all hunt load runs IN ClickHouse (the queries + coordination), so the pod itself is rarely resource-bound. Its real ceiling is architectural (how many hunts one pod can submit and coordinate concurrently and reliably), not CPU/memory - and the large majority of deployments need exactly one pod. So the default is deliberately simple: - **`huntRunner.enabled` (default `true` on every tier):** deploys `dfe-hunt-runner` as ONE steady pod (`replicas: 1`, no ScaledObject at all). This is the resting state for the large majority of deployments - a single always-on pod, not scale-to-zero. - **`huntRunner.autoscaling.enabled` (default `false`, BETA, keep it off on 2.2.2 - see the top of this page):** opts into the ScaledObject described above. `replicas` is then omitted from the Deployment (KEDA owns the count) and the pod count floats between `minReplicaCount` (default **1**, not 0 - set it to `0` explicitly via values to additionally opt into scale-to-zero) and `maxReplicaCount` (default 5), driven by the due backlog. Work still divvies up safely across N pods via the existing `hunt_lease` mutual exclusion - KEDA picks the pod count, the leases decide who does what, so nothing double-runs. - **Disabled (operator choice):** set `huntRunner.enabled: false` in the deploy overlay. No pod and no ScaledObject are rendered, so hunts do not fire. Re-enable by restoring the dial - the schedule + watermarks are still in ClickHouse, so the runner resumes cleanly from the last committed window. Either way the mechanism is a values dial, turned through the gitops overlay, never a live cluster mutation.