# Autoscaling: the two layers and the per-target fork

> Two independent layers scale a DFE deployment. Keep them apart:

```mermaid
flowchart LR
    subgraph pods["Pod scaling (identical everywhere)"]
        KEDA["KEDA ScaledObjects\n(queue depth, scaling_pressure,\nhunt backlog via dfe-keda-shim)"]
    end
    subgraph nodes["Node scaling (FORKS per target)"]
        CA["cluster-autoscaler"]
        KARP["Karpenter"]
        NAP["GKE NAP"]
        STATIC["static node pool"]
    end
    KEDA -->|"pods go Pending"| nodes
```

- Pod scaling is KEDA, on every target. The charts ship the ScaledObjects (receiver, loader, hunt-runner via the fail-safe dfe-keda-shim). Nothing about KEDA changes between on-prem and cloud.
- Two triggers render and the HPA takes the higher. The app charts ship both `keda.cpu` (native scaler, nothing of ours in the path) and a `metrics-api` entry in `keda.extraTriggers` (`argocd/values/apps/_fragments/pressure-trigger.yaml`), which is the scalo ScalingPressure composite through the fail-safe dfe-keda-shim. A tier or an instance drops it by setting `keda.extraTriggers: []`.
  - The shim matches the bare `scaling_pressure` gauge and any `<prefix>_scaling_pressure` for that ServiceName, so the four wire names the Rust apps use today all resolve without an app release.
  - The shim address is derived from the release namespace, so a deployment in any namespace resolves it without a values override. `keda.pressure.shimAddress` is for a shim outside the release namespace only.
  - dfe-transform-vector carries no pressure trigger. It sets no `scaling_pressure` gauge, so cpu is its only live trigger.
  - `bootstrap/keda-scale-test.sh` proves the path on each deploy. It injects pressure for a throwaway Deployment, then for every app whose ScaledObject carries the shim trigger at once, and each goes from its floor to one above and back. Its scale-out bound is the HPA sync period twice over plus a pod-start allowance, because above a floor of zero the HPA reads the metric itself and KEDA's pollingInterval doesn't govern scaling. At scale-out it logs each HPA reading and fails when the pressure reading is under target, so a cpu-driven scale can't pass for a pressure one. It also fails an app that wrote no `scaling_pressure` rows of its own, because the shim reads no rows as 0 and that app scales on cpu alone. UNPROVEN (exit 3: the HPA read the pressure over target and left the replicas alone, or an app was off its floor before the run) fails the smoke run like a FAIL does.
- Node scaling forks by target. KEDA makes pods Pending, and what turns Pending pods into new nodes depends entirely on where the cluster runs. That fork is a DECLARED decision, recorded here, never an assumption baked into a chart.

## Which trigger each tier gets

> **Coming from Splunk or Elastic?** DFE scales pods on a composite pressure signal beside the CPU trigger (queue depth, `scaling_pressure`, hunt backlog). The composite gates to 0 when the downstream sink's circuit is open and forces 100 over the memory threshold, which CPU alone can't see.

CPU at 70% is the baseline everywhere and never comes out. It's native KEDA with nothing of ours in the path, so a component outage can't take scaling with it. Each tier decides whether the composite renders beside it.

| Tier | Pressure | Ceiling | Why |
| ---- | -------- | ------- | --- |
| slim | off | 2 | No broker and a 2-replica ceiling. CPU is the whole signal worth having, and the shim isn't worth putting in the path for it. |
| single | on | 3 | Carries a broker, so `kafka_lag` measures the backlog an I/O-bound loader's CPU hides. Ceiling = `defaultTopic.partitions` (3), because a 4th consumer joins the group and gets nothing. |
| mesh | on | 10 | No broker, but every stage holds what the next hasn't taken, so `buffer_depth` and `memory` carry what `kafka_lag` carries elsewhere. Replicas share load through the Service, so no partition count bounds the ceiling. |
| scale | on | 12 | Carries a broker. The ceiling is a DIVISOR of `defaultTopic.partitions` (12) so every reachable replica count holds the same number of partitions. At 10, two consumers hold 2 and eight hold 1, and the busy pair sets the group's lag. |

Why the composite pressure and not KEDA's native `kafka` scaler? The native scaler reads consumer-group lag from the broker and needs no shim. It's also blind to the two things that decide whether another pod helps. The composite gates to 0 when the downstream sink's circuit is open (more loaders can't relieve a dead ClickHouse) and forces 100 over the memory threshold (scale before the OOM). Lag alone would scale the loader out against a sink already refusing writes, and the composite carries the lag term too. A deployment that wants the native scaler can have it: `keda.triggers` takes a verbatim scaler list the chart passes straight through.

The thresholds are inherited, not measured. 70 on both the CPU trigger and the 0-100 composite came across from the fleet default and has no load run behind it. Sizing them needs a run with more source topics or more partitions on `default_land` than a single-source rig has, because replicas 4-12 would otherwise idle.

## The fork decision for DFE deployments

> **Coming from Splunk or Elastic?** Node scaling isn't one mechanism here. KEDA scales the pods on every target, and the node autoscaler is chosen per target, with a static node pool on-prem because Karpenter needs a cloud provisioning API.

| Target | Node autoscaler | Notes |
| ------ | --------------- | ----- |
| On-prem Rancher/RKE2 (the baseline) | **None today: static node pool.** cluster-autoscaler IF the node layer gains a provisioning API (Rancher node pools on a cloud provider, Harvester, etc.) | Karpenter is NOT an option here -- it needs a cloud provisioning API. A fixed node pool is the sovereign/air-gap default: capacity planning replaces node autoscaling, and KEDA still does the pod side. |
| AWS EKS | **Karpenter**, the node-pools chart's `karpenter-aws` renderer | The one renderer built today. |
| Azure AKS | **Node Auto Provisioning** (managed Karpenter) | A planned renderer, dfe-infra#806. |
| GCP GKE | **ComputeClass plus OpenTofu node pools** | A planned renderer, dfe-infra#806. GCP has no supported Karpenter. |
| Multi-cloud / mixed | **cluster-autoscaler** | The portable fallback. Runs everywhere. |

One rule covers every component whose best implementation differs by target (node autoscaler, gateway controller, secret store, load balancer, storage class): state the choice per target, keep the chart neutral with the choice in values or the target overlay, and name the portable fallback (here cluster-autoscaler) so no target is stranded.

## Where the choice lives

Node scaling is a pools interface. The resolver writes one pool class per workload, in fields every node autoscaler shares, and `helm/charts/node-pools` hands the classes to the renderer `pools.renderer` names. Karpenter on AWS (`karpenter-aws`) is the one renderer today. The AKS and GKE renderers are planned in dfe-infra#806.

`argocd/appsets/layer2-platform.yaml` deploys the pools chart only to a cluster whose secret declares a cloud with a renderer (AWS today), so the on-prem baseline gets no pool object at all. Every other chart places a pod by the pool's `dfe.hyperi.io/workload` label and the cloud-neutral node labels, never by Karpenter's or a cloud's own (`scripts/tests/test_charts_select_only_the_pool_label.py`).