Deployment portability matrix
DFE is ONE product deployed into many estates. Every major component sits behind a config-driven seam so a deployment swaps it without a code change -- the seam is always an existing k8s abstraction or a chart mode, never a fork. This page is the per-target instantiation of that contract. The seam DEFINITIONS (the dial each one turns) live in ../architecture.md "Swappable components" and stay the SSoT.
RKE2 on-prem is the STANDARD and the tested path. The managed-cloud columns are overlays: same charts, same seams, different values + credentials. A cloud column is not built until the on-prem path is green, but the SEAM must be clean now so the overlay is config, not rework.
The invariant core vs the swappable edge
One cloud L4 LoadBalancer backs the single in-k8s Envoy Gateway, which does ALL L7 routing -- there are no per-app cloud LBs and no cloud L7 (ALB/App Gateway). dfe-engine (the identity authority), the Envoy edge, and HyperDX's DocumentDB store are always in-cluster; Kafka, ClickHouse and object storage can be in-cluster or a managed endpoint.
The matrix
| Component | Seam (the dial) | On-prem RKE2 | AWS | Azure | GCP |
|---|---|---|---|---|---|
| Kubernetes | vanilla-cluster contract (preflight) | RKE2 (standard) | EKS | AKS | GKE |
| External DNS | external-dns provider / estate record | estate CoreDNS record | Route53 | Azure DNS | Cloud DNS |
| Secrets | ESO ClusterSecretStore | OpenBao | Secrets Manager | Key Vault | Secret Manager |
| Private CA / TLS | cert-manager ClusterIssuer (+ external issuer) | OpenBao PKI | ACME or AWS PCA | ACME or Key Vault | ACME or Google CAS |
| Kafka | chart mode + kafka.bootstrapServers | Strimzi / Redpanda* | MSK / Confluent Cloud / Redpanda Cloud, each on AWS** | Confluent Cloud on Azure** | Confluent Cloud / Redpanda Cloud, each on GCP** |
| ClickHouse | clickhouse.mode + .host | in-k8s operator | CH Cloud (pref) / in-k8s | CH Cloud (pref) / in-k8s | CH Cloud (pref) / in-k8s |
| Edge LoadBalancer | envoyGateway.service type=LoadBalancer | MetalLB | NLB | Azure LB | GCP LB |
| Object store | S3-compatible endpoint + creds | MinIO / Ceph RGW | S3 | Blob (S3 API) | GCS (S3 API) |
| Container registry | global.registry, plus a pull secret for a private mirror | Harbor | ECR | ACR | GAR |
| Image pull cache | containerd mirror per upstream | not built | ECR pull-through cache | ACR artifact cache (not built) | AR remote repos (not built) |
| StorageClass / CSI | storageClass (empty -> cluster default) | local-path / Longhorn | EBS | Azure Disk | PD |
| Workload identity | ServiceAccount annotations | static creds (OpenBao) | EKS Pod Identity (no annotation) | Azure Workload Identity | GKE Workload Identity |
| etcd-at-rest KMS | cluster EncryptionConfiguration provider | secretbox / aescbc | KMS | Key Vault KMS | Cloud KMS |
| Document store | documentdb chart, a DocumentDB CR on the DocumentDB operator's CloudNativePG (always in-cluster, Kubernetes 1.35+, or 1.34 with the ImageVolume feature gate) | 2 instances, gateway in each | " | " | " |
| Identity authority | dfe-engine account store (always in-cluster) | dfe-engine | dfe-engine | dfe-engine | dfe-engine |
| Federated login | oidc.providers external OIDC (opt-in) | any OIDC IdP | Entra / Okta / ... | Entra | Google / ... |
* Redpanda on-prem is behind a licence gate.
** A managed backend is offered only as its instance on the deployment's own cloud, and the plan refuses any other: managed-backends.md.
Notes that matter
- One LB, dumb L4. The only cloud LB per deployment is the
Service type=LoadBalancerfronting the gateway (MetalLB provides it on-prem). Cloud L7 (routing, WAF, OIDC) is NOT used -- Envoy owns L7 in-cluster, so behaviour is identical everywhere. Cloud-specific LB tuning ridesenvoyGateway.service.annotations. - Workload identity is the credential seam for cloud. ESO, external-dns and the cloud LB controller reach their cloud APIs via IRSA / Workload Identity, annotated onto the ServiceAccount. On-prem there is no cloud API -- creds come from OpenBao through ESO. Nothing hardcodes a static cloud key.
- etcd-at-rest is a cluster seam, not an app one. dfe-engine (and every
secretsconsumer) inherits whatever the cluster's EncryptionConfiguration uses -- cloud KMS on managed clusters, secretbox/aescbc on-prem. No app couples to a KMS. - Data services are in-cluster OR managed, per deployment. Kafka and ClickHouse
default to in-cluster (Strimzi, CH operator) and swap to a managed endpoint by
pointing the chart's
*.host/bootstrapServersat it. The DocumentDB store's PostgreSQL stays in-cluster (it is HyperDX's store, not a general PG). - ClickHouse on cloud: ClickHouse Cloud is PREFERRED, but self-hosted ClickHouse
in-k8s (the operator) is equally ACCEPTABLE on any cloud -- the
clickhouse.modeseam supports both everywhere, so it is a per-deployment call, not a hard rule. - Nothing cloud-specific is assumed by a chart. A component reachable through only one provider is a bug (architecture.md). The 443/6443 API-egress fix and the local-signer-over-OpenBao correction are the kind of on-prem assumption a de-hardcoding audit exists to catch.
- Choosing an external service changes the cluster's network posture, on
purpose. The baseline is default-deny egress, so
network-policiesderives an egress allow for each external dependency the deployment DECLARES --clickhouse.mode: externalopens the ClickHouse clients on the ClickHouse ports,kafka.mode: externalthe Kafka clients, and likewise for an external OIDC issuer, secret backend, index server or OTel destination. There is no separate acknowledgement flag and there will not be one: the declaration IS the decision. The trade is accepted -- pointing the loader at ClickHouse Cloud widens egress with no second choice made anywhere -- because the alternative is a pod that cannot reach the datastore it was just pointed at, crashlooping with nothing in the policy set to explain why. Each purpose gets only the ports that dependency speaks, and any purpose pins to a known endpoint CIDR (externalEgress.<purpose>.allowCIDRs) where one is fixed.
Node autoscaling
A pools interface, with one renderer per cloud. The resolver writes a pool class per workload, and helm/charts/node-pools renders the classes for the cluster's node autoscaler: Karpenter on AWS today, and AKS Node Auto Provisioning and GKE ComputeClass plus OpenTofu node pools in dfe-infra#806. On-prem runs a static node pool and renders no pool object. ../AUTOSCALING.md has the per-target table.
Status
DESIGN ARTIFACT for review (2026-08-18). The de-hardcoding CODE AUDIT across charts + bootstrap, and the actual EKS/AKS/GKE overlays, are the deferred follow-up -- to be sequenced after this matrix is signed off.