Skip to main content

Deploying DFE with dfe-infra

Bare cluster in, running DFE out: bootstrap lays the Layer 0 baseline, DFE's own Argo CD reconciles the backing services (Layer 1) and the apps + engine-authored overlays (Layer 2). Tier selection and every default live in values files - a deployer overrides single dials, never templates.

The three layers​

  • Bootstrap (bootstrap/bootstrap.sh): probes each dependency - adopt a healthy existing controller, install a DFE-owned copy otherwise. The Argo CD it installed (release argocd in namespace argocd) is upgraded in place on every re-run, so a chart pin or an install flag reaches a live cluster; an adopted Argo is never reconfigured.
  • Layer 1 (argocd/appsets/layer2-data.yaml + layer1-addons.yaml): backing services from base charts + the deploy repo's infra/ overlay.
  • Layer 2 (argocd/appsets/layer2-apps.yaml): one Application per deploy-repo values/*-values.yaml: the component's thin chart over OCI, pinned by digest, beside helm/charts/dfe-extras, with the $values overlay layered last.

Tiers and the values cascade​

The cluster secret's dfe.hyperi.io/profile label selects the tier; values merge in this order (later wins):

<chart>/values.yaml                  chart defaults (helm/charts/, or the app's thin chart)
argocd/values/common.yaml            fleet-wide overrides
argocd/values/<cloud|site>.yaml      per-target overrides
argocd/values/profile-<tier>.yaml    tier composition (slim / single / scale / mesh)
deploy repo sizing/<tier>.data-charts.values.yaml   resolver sizing, layer2-data only
deploy repo infra/common.yaml        deployment-wide (every appset)
deploy repo infra/<chart>.yaml       one data-layer or platform chart
deploy repo values/<app>-...yaml     engine-authored overlay (Layer 2 apps)

Everything from infra/ down is the DEPLOYER's, and it is last, so clickhouse.mode, kafka.mode and the storage models are reachable without editing this repo. The profile file is a tier DEFAULT, not a lock.

The data layer therefore depends on the deploy repo resolving. On a bundled deploy (no external git) the git host is layer2-deploy-repo's own single-source Application, so the data-layer and platform Applications report ComparisonError until it is up and then converge.

What each tier is proven to do:

TierTransportStatus
single, scaleKafka between every stageSupported and proven. They need Kafka: a broker in the cluster, or a managed one (managed-backends.md).
slimgRPC, stage to stage, one replica eachSupported and proven without archiving. A source with archive on routes a copy to the archiver as a second gRPC destination, and that fan-out is not yet proven.
meshgRPC through a balancer per stage pool, no KafkaPresent in the charts, never run against a live cluster, and not supported in 2.2.1. The work to make it a tier is tracked in #875.

gRPC itself is proven app to app: receiver to loader, and receiver to transform to loader, on the docker stack's *-grpc profiles and on slim.

Kafka on tier single uses the non-operator single-broker KRaft path (helm/charts/kafka, kafka.mode: single); the Strimzi operator installs only on scale clusters (layer-scale.yaml).

Secrets store​

TierReads the secrets storeWhat to set
slim, single, meshnothingnothing: DFE_SECRETS_BACKEND resolves to none, no ClusterSecretStore
scalethe Strimzi broker's client credentialDFE_SECRETS_BACKEND=openbao with DFE_VAULT_ADDR, DFE_VAULT_ROLE_ID, DFE_VAULT_SECRET_ID, or aws-sm with DFE_SECRETS_REGION

dfe-ops stack-deploy checks each store-referencing template's render condition and refuses none where one fires. DFE_CA_PERSIST=true, gateway OIDC (on in aws.yaml), an external or managed broker, an external ClickHouse and object storage each add a consumer on any tier. A deploy-repo overlay can too, unseen before the deploy. On AWS the backend is aws-sm and openbao is refused; elsewhere an unnamed backend is openbao, as is any OpenBao value (managed-backends.md).

Every DFE image and thin chart pulls anonymously from DFE_REGISTRY, so no pod names a pull secret.

Which APPS a tier deploys is apps.yaml's, not the profile file's -- composition.md has the table and the derivation.

The mesh tier​

Not supported in 2.2.1: the pieces below render and pass their chart tests, but no mesh deploy has run against a live cluster (#875). mesh is scale without a broker: the stages hand records to each other over gRPC. mesh.enabled puts a Gateway API listener in front of every stage pool, because a Kubernetes Service balances per connection and gRPC holds one open, so a sender would otherwise pin itself to one pod however many replicas KEDA adds. Each pool's route carries its own policy (mesh.routePolicy): a request timeout at or above the sender's 30 s deadline, retries only on connect failure and resource exhausted, and ejection of a pod that keeps answering unavailable.

With acknowledgements.enabled (the default) the receiver answers a sender only once the loader confirms delivery, and builds no buffer or spool. A loader outage reaches senders as a 503 once the hold expires, and they retry from their own copy. With acknowledgements off, the raised receiver buffer is the only slack in the chain: an outage shorter than it is invisible to senders, and a held record has already had its 2xx, so it dies with the pod.

Storage model - decided at deploy, not after​

Both data stores take a storageModel that fixes the on-disk layout for the life of the deployment. It defaults to local, which changes nothing. Models are named <family>-<bulk>: the family says whether data MOVES to the bulk store or is COPIED to it, the bulk half says what that store is.

chartlocalthe alternatives
clickhouse-clusterparts on the PVCcached-object - parts on an object-store disk behind a local read-through cache; tiered-block - a hot SSD volume and a cold bulk volume, parts demoted as the hot one fills
kafkaevery segment on the PVCtiered-object - closed segments to object storage (KIP-405 tiered storage); the PVC sizes the hot window

Growing a PVC instead needs allowVolumeExpansion: true on the StorageClass and a StatefulSet recreate, because volumeClaimTemplates are immutable. local-path cannot resize at all. That is what the storage model exists to avoid.

Every model with an object bulk store reads its credentials from the environment, materialised by ESO from <project>/<env>/clickhouse/s3 and <project>/<env>/kafka/tiered - never inline in a values file.

The engine holds the models and the modes as protected vars, so the API refuses a post-deploy edit. See the deploy repo's infra/README.md.

Every combination, its status and its evidence: storage.md - the deploy-time matrix, including the cells that are refused and the ones not built yet.

Upgrading onto the generated JWT signing key​

Releases before the ESO-generated key minted dfe-engine-jwt from the chart template. On the first upgrade past that, the ExternalSecret adopts the existing Secret (creationPolicy: Owner) and writes a new key into it, so every token issued before the upgrade stops verifying and the engine rolls once - operators and any machine caller log in again, and nothing else is affected. refreshPolicy: CreatedOnce then holds that key for the life of the deployment: on the rc.12 deployment the Secret's resourceVersion moved on that first reconcile and held across the two renders after it. A deployment that cannot take even one invalidation sets auth.jwtSecret to the key it already has, which renders a plain Secret and no generator.

Upgrading onto the main landing table​

The landing table and the catch-all source are both named main. A deployment cut before that rename landed has a default table holding its rows, and nothing moves or drops it: the engine's DDL writer creates main alongside it, the receiver stamps an unmatched record _source: main, and the loader writes new records to main. Query the old table directly for anything older than the upgrade, and drop it once nothing needs it.

The Kafka landing topic moves with the source name, from default_land to main_land. dfe-engine creates the new one at startup. The old topic keeps whatever it already holds until it is deleted.

Version pins​

versions.yaml is the single source for chart, operator, and image versions, checked by scripts/check_versions_drift.py. Pin rule for images: name:tag@sha256 digests - tags live in apps:, the immutable digest half in digests: (scripts/dfe-stack renders the combined form and dfe-stack verify re-checks digests against GHCR). A dfe-infra release tag certifies the whole set as one stack version (stack: metadata + lockstep content: repo tags); the full release model is in dfe-docs deployment/stack-versioning.md.

content: also records the dfe-docker ref that runs each stack. dfe-docker cuts no per-stack tag, so the entry is a commit rather than a version, and dfe-stack cut re-resolves it from dfe-docker main at cut time. To reproduce an old stack's docker path, check dfe-docker out at that stack's recorded ref; for the current stack, main is the ref.

The engine also serves authored files an app ships - the reference transform pipelines and each app's config contract. The engine chart's content.entries turns each into one init container that fills /etc/dfe-engine/content from the pinned app image, and the engine reads that directory through DFE_LIBRARY_SEED_DIR and DFE_APP_CONTRACT_DIR. The files never travel through a value or a ConfigMap, because either form re-serialises them into etcd on every Argo sync. The source catalogue the console offers is not one of them: it is dfe-schemas package data, installed with the engine.

Version check​

Every DFE service boots with an opt-out release check: one POST of {product, current_version, os, arch, instance_id} to the HyperI releases endpoint, logging whether a newer version exists. The id is a one-way UUIDv5 derived from the platform; deployment names are never sent; any failure costs one WARN line and nothing else.

The versionCheck values key is the deployment override, rendered into every app by the dfe-common.versionCheckEnv helper -- fleet-wide in argocd/values/common.yaml, or per app in its overlay:

versionCheck:
  enabled: false        # total opt-out -- no check, nothing sent (air-gap)
  sendInstanceId: false # keep the check, strip the install id
  apiUrl: ""            # point the check at a mirror

Optional: your own private mirror​

For a mirror that needs a login, point DFE_REGISTRY at it and set DFE_PULL_SECRET_TOKEN, with DFE_PULL_SECRET_USER (default token) and DFE_PULL_SECRET_SERVER (default DFE_REGISTRY's host). bootstrap creates ghcr-pull-secret in every DFE namespace, records it on the cluster secret (dfe.hyperi.io/image_pull_secret) for the appsets to hand their charts, and registers the token as Argo CD's read credential for each thin chart.

  • architecture.md - where this repo sits in the suite
  • wizard.md - dfe-ops init, the prompt-driven wizard that writes a deployment.yaml dial
  • composition.md - which apps a profile deploys by default, how apps.yaml's default_in reaches Argo and Compose, and what an app with nothing to do does instead of crash-looping
  • storage.md - the storage-deploy matrix: service x mode x storage model, with the status and evidence behind every cell
  • edge.md - the edge module: every door into a deployment, grouped and tiered, with one table per flavour of what is on, what it costs and which dial key turns it
  • gateway-oidc.md - edge OIDC: the values that turn it on, the private-CA IdP shape, and what Envoy Gateway cannot do with a groups claim
  • edge-vpn.md - the opt-in tunnel a field appliance dials in on: the two ports, the reserved client range, and how it reaches receivers only
  • toolbox.md - the troubleshooting image, the opt-in in-cluster pod, and the on-demand AWS bastion: off by default, read-only Kubernetes API access, and what each surface records
  • rke2.md - the default distribution
  • on-prem.md - build, deploy, test and tear down on an on-prem cluster through the local target
  • aws.md - deploying on AWS: EKS, Karpenter, sizing, and the MSK/Confluent/Redpanda Kafka and ClickHouse choices
  • aws-operations.md - running a deployed AWS cluster: node pools and images, admin UI exposure, Kafka telemetry and autoscaling, upgrades
  • aws-teardown.md - tearing an AWS cluster down: lifecycle tags, the teardown order, and sweeping what is left
  • upgrade-rollback.md - the rollback runbook: what rolling back means per upgrade stage, the dfe-ops upgrade rollback refusal rule, the sizing config-vs-data rule, and the soak before a Kafka finalise
  • DEPLOY-HELPERS.md - the release and deploy-overlay helpers, the logins a deploy carries, and the end-to-end recipe
  • DEPLOY-TLS-TRUST.md - which CA signs the gateway certificate: the self-signed default, estate PKI, and trusting the root once
  • kafka/ - managed-Kafka alternatives + Redpanda gate
  • clickhouse.md - CH target matrix (official operator / ClickHouse Cloud; Altinity untested) + operator history, including the retired private-cloud pairing
  • upgrades.md - migrating a pre-2.2 private-cloud deployment off the retired ClickHouse fork onto cached-object
  • deployment-logs/ - record every deploy