Skip to main content

Hunt-runner scaling: deterministic-due, no engine dependency

The hunt runner is pull-based and multi-pod. Default deployment is ONE steady pod (no ScaledObject at all) - the runner is a thin orchestrator (all hunt load runs in ClickHouse), so one pod covers the large majority of deployments. Autoscaling is BETA and opt-in (huntRunner.autoscaling.enabled). When turned on, KEDA scales the pod count on the deterministic-due backlog this doc explains, via the fail-safe dfe-keda-shim - never the engine, and no longer a MySQL-on-ClickHouse scaler (retired, see "KEDA ScaledObject" below).

Keep autoscaling off (the default) on 2.2.2. It reads the due backlog from the hunt_schedule table, and nothing in the 2.2.2 charts writes that table: there is no materialise Job, so the backlog always reads 0. With autoscaling on the runner never scales out, and at minReplicaCount: 0 no hunt ever fires. To try it anyway, run dfe-hunt-runner materialise after every hunt-config change.

This doc is the SCALING contract: how the due backlog is computed with zero ambiguity, how the beta ScaledObject wakes/grows/shrinks the pod count from it, and the property that matters most - the runner has NO operational dependency on dfe-engine.

The one hard requirement: due-detection with zero workers​

This is the contract the BETA autoscaling relies on (see "Default deployment, beta autoscaling, and disable" below for when it applies - it also governs correctness if you opt into minReplicaCount: 0, i.e. scale-to-zero). KEDA can only wake the first worker if "is any hunt due right now?" is answerable with no worker running - i.e. as a query KEDA itself (via the shim) runs against ClickHouse. But the per-hunt schedule (interval + stable phase offset) otherwise lives only in a running worker's memory (loaded from the gitops hunt configs). At zero workers there is nobody to ask.

So the schedule is MATERIALISED into a small ClickHouse table, hunt_schedule, at config-deploy time. The scaler then reads it directly. Three tables already exist for coordination (ch_coordinator.py): hunt_lease, hunt_watermark, hunt_state. hunt_schedule is the fourth, and the only one written outside the worker loop.

hunt_schedule(hunt_id, interval_seconds, phase_offset, enabled, updated)
  ReplacingMergeTree(updated) ORDER BY hunt_id

enabled is a soft tombstone: a deleted hunt is written enabled=0 so it stops waking KEDA, without a DELETE.

The scaler query (deterministic-due)​

KEDA runs schedule.due_query(db) against ClickHouse (via the fail-safe dfe-keda-shim's /keda/hunt-backlog endpoint - see "KEDA ScaledObject" below) - the count of hunts that are due-and-unclaimed. It uses the EXACT arithmetic the worker uses (schedule.latest_fire):

boundary = intDiv(now(), interval) * interval      -- start of this interval
current  = boundary + phase_offset                  -- this interval's scheduled fire
fire     = if(now() >= current, current,            -- the latest fire at or before now
              current - interval)
due      = watermark < fire                          -- that fire not already completed
           AND lease_until <= now()                  -- not currently running

due_count > 0 -> scale up, and == 0 -> scale down (to zero only where minReplicaCount: 0 has been opted in - the default floor is 1 pod). The enabled=1 filter is applied in an inner subquery so a tombstone (interval=0) can never reach the intDiv (no divide-by-zero).

Why the watermark comparison is exact​

The worker writes watermark = scheduled_start (the fire time) after a run commits (worker.py: window(...) returns end = scheduled_start, then set_watermark(end)). So the watermark is in the SAME epoch-fire scale as the boundary arithmetic - watermark < fire means precisely "this fire has not been completed". No unit mismatch, no data-time-vs-fire-time skew.

Why it is RELIABLE​

Autoscaling (when the BETA is enabled) is only safe if KEDA's view of "due" never diverges from a worker's. It cannot, because both compute the same thing from the same inputs:

  1. Parity, tested. The scaler predicate equals runner.tick's own gate (via schedule.latest_fire + the watermark + lease checks) across a full matrix of (hunt_id, interval, now, watermark, lease), and the SQL implements that predicate against real ClickHouse - counting exactly the runnable hunts, excluding tombstoned / future-watermark / leased hunts, and returning 0 when every hunt is caught up.
  2. Single source of truth for the offset. The materialiser computes phase_offset with the same schedule.phase_offset the worker uses, so the two can never drift. Change the offset algorithm once and both move together.
  3. Gate on the signal, not a timer. KEDA polls the backlog query (via the shim - a real readiness signal), it does not race a clock. Worst-case wake latency is pollingInterval + pod-start. For hunts (minute-plus intervals) that is immaterial. This is the "gate on the real condition" rule from the testing standard applied to autoscaling.
  4. Self-healing. If a worker claims a hunt then dies before writing the watermark, its lease expires (lease_seconds, default 300) and the fire is still owed (watermark < fire), so the scaler keeps >=1 worker and the hunt is reclaimed. Nothing double-runs (a hunt run twice writes and notifies twice): a claim stands only if acknowledged within a settle of its free read and still the latest row after it. A later claim is never released, as it may sit over the winner, but it holds nothing, so the next poll claims again.
  5. No thundering herd. The phase offset spreads same-interval hunts across the interval, so due_count rises gradually and KEDA scales out smoothly rather than all-at-once at the boundary. The offset falls in the first hunts.phase_spread_fraction of the interval (default 0.8, env DFE_ENGINE_HUNTS_PHASE_SPREAD_FRACTION). The runner, the API's next_due and the materialised schedule all read it, so republish the schedule (dfe-hunt-runner materialise) after changing it.

Bounded catch-up: a fire stays owed until its watermark is written, so a worker that ticks inside [boundary, boundary+offset) runs the previous interval's fire if nothing ran it, and the scaler counts it the same way. Without that, a hunt whose offset sits between two poll ticks (a 46s offset polled at :00/:15/:30/:45) never fires at all. Max lateness for any fire is under one interval - inherent to the phase-offset design, and identical whether the decision is made by KEDA or a resident worker.

No operational dependency on dfe-engine​

This is the point of the pull-based design. Once deployed, the running system needs only ClickHouse:

  • Workers read hunt defs from the gitops config (mounted / cloned), coordinate through the four CH tables, and execute queries against CH. No engine call on the hot path.
  • KEDA reads hunt_schedule + the coordination tables from CH via the dfe-keda-shim - a separate, distinctly-labelled pod running the SAME engine container image under a different entrypoint (dfe-keda-shim run, not the live dfe-engine API service). No call to the running engine API.
  • The engine's only role is at CONFIG-DEPLOY time, not runtime: materialise hunt_schedule when the hunt config changes, with the one-shot, idempotent dfe-hunt-runner materialise. The 2.2.2 charts do not run it (see below).

So if the engine pod is down, hunts keep firing, and so does KEDA scaling once the schedule is materialised. The only thing you lose while the engine is down is the ability to CHANGE configs - which needs the engine anyway. ClickHouse is DFE's single operational store, and the runner holds to that (no Postgres, no engine, no convenience store on the hot path), which is also what lets the same runner work on a non-k8s single deploy (dfe-docker) where only ClickHouse is guaranteed present.

Coordination needs a SINGLE LOGICAL ClickHouse (scale-tier requirement)​

The four coordination tables are the shared source of truth every worker and KEDA read/write. That only works if all of them see the SAME tables. On the scale tier's multi-node cluster behind a round-robin load balancer, a plain ReplacingMergeTree (and a plain CREATE DATABASE) would land on one node, so workers whose connections land on other nodes would see an empty or absent table and coordination would silently break (UNKNOWN_DATABASE, missing leases). The schema phase therefore creates the coordination tables as ReplicatedReplacingMergeTree inside the Replicated dfe database on every deployment, so lease, watermark and schedule state is shared across nodes (https://github.com/hyperi-io/dfe-schemas/blob/main/docs/clickhouse-platform.md).

Materialisation (config-deploy time)​

schedule.publish_schedule(ch, db, specs) is the ONLY writer of hunt_schedule, and dfe-hunt-runner materialise is its only caller. It is idempotent and diff-aware (tombstones hunts that vanished from config). The 2.2.2 charts run no Job for it, so on Kubernetes the table stays empty unless an operator runs the command after each hunt-config change. Run that way, a brand-new hunt added while scaled to zero still gets a hunt_schedule row, so KEDA can wake a worker for it.

KEDA ScaledObject (shape)​

KEDA has no native ClickHouse scaler. The earlier approach - KEDA's mysql scaler pointed at ClickHouse's MySQL-compatible interface (mysql_port, 9004) - has been RETIRED: KEDA's only failure knob for a DB scaler is a fixed fallbackReplicas count, not "hold at current", so a broken metric could scale the wrong way and blow cloud spend. It is replaced by a small OWNED, fail-safe adapter: the dfe-keda-shim, its own pod on the engine image (dfe-keda-shim run), labelled so the engine Service never routes to it. The published engine chart (oci://ghcr.io/hyperi-io/charts/dfe-engine) renders only the engine pod. The shim comes from dfe-infra helm/charts/dfe-extras/templates/engine-keda-shim.yaml, on the same values. KEDA's metrics-api scaler polls the shim over HTTP, and the shim runs schedule.due_query(db) against ClickHouse - the SAME SQL the worker gate uses (see "The scaler query" above, imported directly from dfe_engine.hunts.schedule.due_query, so it cannot drift) - and hands back one integer.

Ships with the hunt-runner Deployment, gated by huntRunner.autoscaling.enabled (default false, BETA). When enabled, the Deployment's replicas field is OMITTED (the ScaledObject owns the count instead) and a dfe-hunt-runner-scaler ScaledObject is rendered alongside it (dfe-infra helm/charts/dfe-extras/templates/engine-hunt-runner.yaml):

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: dfe-hunt-runner-scaler
spec:
  scaleTargetRef:
    name: dfe-hunt-runner
  minReplicaCount: 1     # default 1, NOT 0 - set 0 via values to opt into scale-to-zero
  maxReplicaCount: 5     # default; huntRunner.autoscaling.maxReplicaCount
  cooldownPeriod: 300
  pollingInterval: 30
  triggers:
    - type: metrics-api
      metricType: AverageValue     # ceil(due / targetBacklog) workers
      metadata:
        targetValue: "10"          # huntRunner.autoscaling.targetBacklog
        # huntRunner.autoscaling.shimAddress, else the shim in the release's own namespace
        url: "http://dfe-keda-shim.<release-namespace>.svc.cluster.local:8080/keda/hunt-backlog"
        valueLocation: "value"

The fail-safe (dfe_engine/keda_shim/shim.py, QueryShim.run): every query caches its last-good integer per (query, params). On ANY failure - CH unreachable, timeout, a bad query - the shim returns the CACHED value, so KEDA sees no change and the pod count FREEZES at current, so it can never run replicas up on a broken metric. A cold start with no cache yet, plus a failure, falls back to a conservative cold_hold (0 for the backlog query - hold at minReplicaCount). This cache-or-cold-hold behaviour is exactly why the shim replaced the zero-code MySQL-on-ClickHouse route: KEDA's native DB scalers cannot express "hold at current", only a fixed fallback count.

Other notes: the shim is a separate, distinctly-labelled pod (app.kubernetes.io/name: dfe-keda-shim) running the same engine container image under a different entrypoint - it is NOT the live engine API, so hunt-runner scaling still has no dependency on the engine service being up. Queries (including the backlog query's clamp/cold-hold/timeout) are config-driven (keda_shim/queries.yaml + an optional mounted override), so a schema change or a tuning change is a config edit, not a shim rebuild.

Proving it: the artificial scale test​

bootstrap/keda-scale-test.sh (dfe-infra) is the mechanised proof that the shim -> KEDA scale-out/scale-in path actually works, rather than an aspirational claim: it forces the path on demand with a disposable target instead of waiting for real hunt load - stand up a throwaway Deployment + ScaledObject wired to the shim's pressure query for a test ServiceName, INSERT a high dfe_scaling_pressure row into otel_metrics_gauge (any <prefix>_scaling_pressure spelling matches - see keda-scaling), poll (bounded, 120s) for scale-OUT to >= 2 replicas, drag the rolling average back down, poll (bounded, 180s) for scale-IN to 1, then self-clean. It rides the SAME shim endpoint and metrics-api mechanism the hunt-runner's /keda/hunt-backlog trigger uses, so a pass proves the mechanism the hunt-runner beta relies on, not a mock. Wired into bootstrap/run-all-smoke-tests.sh (gated DFE_KEDA_SCALE_TEST, default on).

Default deployment, beta autoscaling, and disable​

The hunt runner is a thin orchestrator - virtually all hunt load runs IN ClickHouse (the queries + coordination), so the pod itself is rarely resource-bound. Its real ceiling is architectural (how many hunts one pod can submit and coordinate concurrently and reliably), not CPU/memory - and the large majority of deployments need exactly one pod. So the default is deliberately simple:

  • huntRunner.enabled (default true on every tier): deploys dfe-hunt-runner as ONE steady pod (replicas: 1, no ScaledObject at all). This is the resting state for the large majority of deployments - a single always-on pod, not scale-to-zero.
  • huntRunner.autoscaling.enabled (default false, BETA, keep it off on 2.2.2 - see the top of this page): opts into the ScaledObject described above. replicas is then omitted from the Deployment (KEDA owns the count) and the pod count floats between minReplicaCount (default 1, not 0 - set it to 0 explicitly via values to additionally opt into scale-to-zero) and maxReplicaCount (default 5), driven by the due backlog. Work still divvies up safely across N pods via the existing hunt_lease mutual exclusion - KEDA picks the pod count, the leases decide who does what, so nothing double-runs.
  • Disabled (operator choice): set huntRunner.enabled: false in the deploy overlay. No pod and no ScaledObject are rendered, so hunts do not fire. Re-enable by restoring the dial - the schedule + watermarks are still in ClickHouse, so the runner resumes cleanly from the last committed window.

Either way the mechanism is a values dial, turned through the gitops overlay, never a live cluster mutation.