# DFE on AWS > Operations: [aws-operations.md](/v/2.2.1/dfe-infra/deployment/aws-operations.md). Teardown: > [aws-teardown.md](/v/2.2.1/dfe-infra/deployment/aws-teardown.md). DFE's AWS path provisions an EKS cluster with OpenTofu, then hands it to the same bootstrap and Argo CD layers every other cluster uses. Nothing about Layer 1 or Layer 2 changes for AWS -- the difference is in what `terraform/environments/aws` builds and what the deployment dial resolves. A few things hold for every deployment: - ARM only. Every node pool, and the MSK broker shape when one is built, resolve to Graviton -- the cluster module refuses a `shape_ref` whose resolved architecture is not arm64. - Nodes and the data plane sit in private subnets, behind one NAT gateway by default (`network.nat: single`) or one per availability zone (`per-az`) when a zone failure or the cross-zone charge outweighs the extra gateway. - `network.az_count` (default 3, 2-6) sizes the VPC and is one of the six fields `sizing.yaml` locks once resolved (see [Sizing dials and locks](#sizing-dials-and-locks)). A Strimzi/MSK broker count steps up to at least the AZ count, so a deployment spanning more zones than the tyre-kick floor's three brokers still spreads one broker per zone. - The Kubernetes API's private endpoint is always on. `endpoint.public: true` adds the public one, and only alongside `endpoint.allowed_cidrs` -- the plan refuses a public endpoint with no allow-list. - A gateway VPC endpoint keeps S3 traffic off the NAT path at no extra charge. - The account's state bucket is a one-off: run `terraform/environments/aws-state` once, before any deployment applies. - Every controller that needs an AWS credential -- the EBS CSI driver, Karpenter, the AWS Load Balancer Controller, external-dns, cert-manager -- gets it through EKS Pod Identity, associated to its own service account by the cluster module. Nothing here carries a static key. - external-dns publishes only what the gateway chart marks. Its sources include Gateway API routes, and the Gateway admits routes from every namespace, so without a filter any namespace-scoped actor could attach an HTTPRoute to the wildcard listener and have a name published under your domain with a public certificate behind it. The addon carries `annotationFilter: dfe.hyperi.io/publish-dns=true`, and `helm/edge/gateway` writes that annotation on every route it renders and on the managed proxy's own Service. A route from a deploy-repo overlay is published only if it carries the same annotation. - external-dns runs `policy: sync` with a per-deployment `txtOwnerId` (`argocd/appsets/layer1-addons.yaml`), so a deleted Service or Ingress has its record actively removed -- the chart's own default, `upsert-only`, never deletes anything it published. That covers every teardown except `tofu destroy` itself: the EKS cluster and every node go down in the same run, so whatever external-dns last published is still in the private zone with no controller left to react. `terraform/modules/kubernetes-cluster/ aws/dns.tf` carries a destroy-time cleanup for exactly that case -- it empties the zone of everything but its own apex NS/SOA pair, using the `aws` CLI directly, before the zone itself is destroyed. It runs under whatever AWS identity `tofu destroy` is already authenticated as, so the one added prerequisite is the `aws` CLI on the machine running the destroy. The cluster module's destroy-time sweep of leaked CNI network interfaces (`eks.tf`) needs it too, and a destroy without the CLI fails. - One file drives all of it: the deployment dial, `deployment.yaml`, copied from `deployment.example.yaml`. ## How costs are described Costs here are T-shirt buckets, never figures. A bucket is relative to the deployment's own compute, never to a currency: XS is negligible beside the cluster's compute, S a small fraction of it, M of the same order as one dedicated node, L of the same order as the whole cluster's compute, and XL larger than the cluster's compute. A pricing model is named wherever it decides whether an option is safe -- hourly, per GB processed, per request, per LCU-hour, per broker-hour -- but never a rate. The scale is deliberate: rates move by region, by account and by discount, so a deployer prices their own account rather than trusting a number written down here. Every edge door is priced on this scale in [edge.md](/v/2.2.1/dfe-infra/deployment/edge.md). ## From the dial to a running cluster ### Sizing the deployment - Copy `deployment.example.yaml` to `deployment.yaml`. Every AWS-relevant field is documented inline, including the worked `cloud_aws:` example block near the bottom. - Estimate `sizing.ingest_gb_per_day` in GB/day, or leave it blank for the tyre-kick floor -- the smallest shape that runs the profile without an out-of-memory kill, with whatever throughput it happens to carry reported rather than targeted. On AWS economy focus with `kafka.provider: msk` the resolver's own live run resolves that floor at the smallest shapes the resolver accepts, S, with the managed brokers the largest single line: `m8g.large` for eks-system, `m8g.xlarge` for general, `r8gd.large` for clickhouse, `m8g.large` for keeper, three `express.m7g.large` for msk-broker, and `c8gd.4xlarge` for ci-burst. With Kafka running in-cluster (`kafka.provider: strimzi`) the same floor loses that line and stays S, with kafka-broker on `m8g.xlarge`: its 8 GiB is more than an `m8g.large` leaves a pod. That is the default combined quorum. `kafka.controller_pool: separate` adds three `m8g.large` controllers. Both buckets come from on-demand Linux pricing in the AWS Pricing API, read 2026-09-13 -- rates drift, and the resolver's own `sizing/.report.md` (written under `--out`) is the current answer, not this paragraph. At the floor the resolver's A4 check WARNS rather than fails: `r8gd.large` and `m8g.large` sustain only 3,600 IOPS / 79 MiB/s on their EBS baseline and can burst above it for 30 minutes a day, while ClickHouse and Keeper write continuously. The floor is the smallest shape that runs, not a throughput target -- set `sizing.ingest_gb_per_day` once you know it, so the resolver sizes for the workload instead of the minimum. ### Resolving sizing artefacts - Run `python3 scripts/resolve_sizing.py --dial deployment.yaml --live --out terraform/environments/aws` (or `--fixtures scripts/tests/fixtures/sizing --cloud aws` against a captured catalogue, no AWS call -- capture one first for a region other than us-west-2 with `resolve_sizing.py capture --region --fixtures `). Every artefact lands under `--out` (default the repo root when omitted): `/shapes/resolved/aws-.json` (the instance-type answer, keyed by region because instance-generation availability is not one worldwide), `/sizing.auto.tfvars.json` at the ROOT of `--out` (`node_pools` and `resolved_shapes`, nothing the root does not declare -- named so, never `.auto.tfvars.json`, so it sorts after `render_dial.py --tofu`'s own `dial.auto.tfvars.json` whatever tier was resolved). The resolver is the ONE writer of both variables: it starts from the dial's own `node_pools.system` block (the always-on group a deployer sizes by hand) and merges the pools it derives on top, and `render_dial.py --tofu` emits neither key at all -- so the two producers never collide on the same tfvars variable. Under `/sizing/`: `.values.yaml`, its `.data-charts.values.yaml` subset, `.report.md` (what was sized, from which ratio, in which cost bucket) and `resolved.yaml` (the baseline the next resolve diffs against) -- plus `.nodes.json` on an on-prem resolve, the demand `bootstrap.sh`'s preflight checks the cluster's real nodes against. - Pointing `--out` at `terraform/environments/aws` is what lands `sizing.auto.tfvars.json` beside `render_dial.py --tofu`'s own `dial.auto.tfvars.json`, so `tofu init` picks up both -- every `*.auto.tfvars.json` in that directory loads, in name order. Copy `/sizing/` into the deployment repo's own `sizing/` and commit it there, so a re-size has something to check itself against: `dfe-ops upgrade --dial` diffs a re-resolve against `/sizing/resolved.yaml`, never against `--out`. layer2-data reads its `.data-charts.values.yaml` under `infra/`. - `bootstrap.sh` reads `/sizing/` from the same place, and seeds the data-charts file into a bundled repo. A resolve run anywhere else has to set `DFE_SIZING_DIR` to that directory. On a cluster running Karpenter, bootstrap REFUSES when `.pools.json` is not there, and the node-pools chart refuses to render with no pool classes rather than prune the cluster's NodePools. ### Sizing dials and locks - `sizing.focus` buys headroom over the sized peak: `economy` (40%), `balanced` (60%) or `performance` (100%). None of the three trade away RF3 or `min.insync.replicas=2`, and none of them decides where the KRaft metadata quorum runs -- that is `kafka.controller_pool` below. - `sizing.overrides` is a deployer's per-workload override block, applied last, over whatever the ratios derived -- and still checked against every storage-cap assertion, so an override is a different answer, not an exemption. - `sizing.yaml`'s `locked:` section names six fields locked once resolved -- `partition_count`, `storage_model`, `kafka_provider` (recorded from the dial's `kafka.provider`), `controller_mode` (combined against separate KRaft controllers, `kafka.controller_pool`), `cloud_token` and `az_count`. The resolver records each in `sizing/resolved.yaml` (`az_count` only when it resolved against a cloud catalogue), so a re-resolve given `--previous /sizing/resolved.yaml` that moves any of them refuses and exits 3 unless you also pass `--migrate`. - Four of the six also have a chart key, and the deployment repo holds those keys as protected vars, so the engine's API refuses an edit to them unless the caller holds `helmvars:override`. `governance/policies/sizing-locks.yaml` covers three: `kafka.sizing.*` (the partition count derives from it), `kafka.controllerPool.enabled` and `cloud`. `storage-layout.yaml` covers the storage model (`clickhouse.storageModel`, `kafka.storageModel`). Neither policy governs a direct git commit. - The Kafka provider and the AZ count are OpenTofu inputs with no chart key, so the resolver's lock is the only one they get. The MSK module refuses any broker shape that is not MSK Express (`terraform/modules/managed-kafka/msk/variables.tf`), and `network.az_count` feeds the VPC's zone list in `terraform/modules/kubernetes-cluster/aws/vpc.tf`. - Above 100,000 GB/day the resolver refuses outright and points at a professional-services engagement rather than a generated profile. ### Applying the plan - Render the tofu inputs with `python3 scripts/render_dial.py --tofu`, which writes `terraform/environments/aws/dial.auto.tfvars.json` from the dial's `target.provision` block. Then, from that directory: `tofu init`, `tofu plan -out=deployment.tfplan`, `tofu apply deployment.tfplan`. `init` reads the state bucket, key and region straight out of the tfvars file, so there is no separate `-backend-config` to keep in step. - The root creates the VPC and its subnets, the EKS cluster and its node groups, the MSK cluster when the dial asks for one, the deployment's KMS key, the Route 53 zones and the Secrets Manager store -- and nothing that runs inside Kubernetes. The deploying principal's EKS access entry uses `data.aws_iam_session_context.issuer_arn`, so an SSO permission set, an assumed role, an instance profile or a Lambda role all work; the entry names the IAM role -- with its `aws-reserved/sso.amazonaws.com//` path for SSO -- not the STS session ARN. - A managed broker builds BESIDE the cluster, not after it. The Kafka module takes the network, the cluster name, the KMS key and the Pod Identity trust policy as explicit inputs, and the key's ARN carries the ordering against the key policy that a broker's first log delivery needs, so nothing holds the broker behind the control plane, the node groups or the addons. The broker is still the longest resource in the deployment, so it sets when the apply ends. - `tofu` writes NO outputs to the state until the whole apply finishes, so `--from-terraform` cannot read them while the broker is still creating and the deploy below waits for the apply to end even though Layer 0 and Layer 1 touch no broker at all. Two applies are the way round it -- one `-target=module.cluster` to land the cluster outputs, then the full apply -- but the broker's own annotations are only written by a bootstrap run that happens after it exists, so that route costs a second bootstrap pass. Take the single apply unless the second pass is worth it to you. - `eval "$(tofu output -raw kubeconfig_command)"` gets you `kubectl`. The driver is `python3 scripts/dfe-ops stack-deploy --stack --mode scale --from-terraform terraform/environments/aws --kubeconfig ` -- the same one the k8s team deploys with. It assembles the bootstrap env from the tofu outputs, sets the stack version, runs the offline pin/drift/render preflight, then `bootstrap.sh`, the readiness gate and the two default end-to-end tests; `--check-only` runs the preflight alone, with no cluster contact. `bootstrap/bridge.py` is the reader it imports, not a command an operator runs. - Set `DFE_VAULT_ADDR` / `DFE_VAULT_ROLE_ID` only when `secrets.backend` is `openbao`. `k8s.repo_url` is the dfe-infra chart repo Argo reads. A private copy needs `DFE_REPO_TOKEN` (an HTTPS token with read access; `bootstrap.sh` turns it into the Argo repository credential), without which every Layer 2 Application stays `Unknown` with "authentication required". Export it, or carry it in an `--env-file`. The dfe-* images and thin charts pull anonymously. - When `k8s.storage_class` names no class the cluster already has, `bootstrap.sh` creates it -- provisioner `ebs.csi.aws.com`, gp3 baseline 3,000 IOPS / 125 MiB/s, encrypted with the account's default EBS key, `WaitForFirstConsumer` -- because EKS 1.30+ ships only `gp2` with no default class and the CSI add-on creates none itself. It is marked the cluster default only when the cluster has none, so an existing default is left in place. Per-use-case classes are a follow-on. The same run's internal-CA persist step renders the gateway chart with the identical value files Argo layers afterwards: `argocd/values/common.yaml`, `argocd/values/.yaml`, `argocd/values/profile-.yaml`. - The AWS Load Balancer Controller is given `vpcId` (the `DFE_VPC_ID` output, carried on the cluster secret as `dfe.hyperi.io/vpc_id`) and `region` explicitly rather than discovering them itself; node IMDS hop limit stays at 1. ## Kafka and ClickHouse Every managed backend on AWS is the vendor's AWS instance: the plan refuses a backend AWS does not offer and a ClickHouse Cloud service on another cloud. [managed-backends.md](/v/2.2.1/dfe-infra/deployment/managed-backends.md) has the rule, the per-cloud table and the egress ports. ### Kafka - `kafka.provider: msk` is the only managed Kafka the AWS root builds today. It runs MSK Express brokers -- Express only, because MSK cannot convert a Standard cluster to Express later -- authenticated both ways: SASL/SCRAM for every DFE app, and SASL/IAM for the in-cluster bootstrap Job alone, since Kafka ACLs live in the data plane and tofu cannot write the first one. That Job writes only the SCRAM principal's ACLs and exits non-zero on any failure. dfe-engine creates the topics. - MSK's broker-count scaler, its telemetry sink and its rebalancing behaviour are operational detail, not part of the provider choice -- see [Kafka telemetry and autoscaling](/v/2.2.1/dfe-infra/deployment/aws-operations.md#kafka-telemetry-and-autoscaling) in aws-operations.md. ### Managed and SaaS Kafka bodies - `confluent-cloud` and `redpanda-cloud` module bodies exist under `terraform/modules/managed-kafka/`, contract-tested like `msk/`, and the AWS root now calls whichever one `kafka.provider` names -- all five tokens validate on both the resolver and the tofu root (`strimzi`, `redpanda`, `msk`, `confluent-cloud`, `redpanda-cloud`), each body behind its own `count`. Neither SaaS body sizes, versions or scales itself from a shape the resolver picks, or creates a topic: dfe-engine creates them. `confluent-cloud` checks `message_max_bytes` against its tier's ceiling. `redpanda-cloud` takes the shared SCRAM password and the ephemeral-lifecycle `allow_deletion` flag. Only `name`, `env` and `network` are true of every body regardless of provider. The vendor API credential is a provider environment variable, never a tfvar: `CONFLUENT_CLOUD_API_KEY` / `CONFLUENT_CLOUD_API_SECRET` for Confluent, `REDPANDA_CLIENT_ID` / `REDPANDA_CLIENT_SECRET` for Redpanda Cloud -- export both before `tofu plan`, since the `redpanda` provider checks for its pair at configure time regardless of which `kafka.provider` is selected. Confluent's default tier (Freight) is a Private Network Interface, not PrivateLink, and caps a topic's `max.message.bytes` at 20 MB, so it carries the chain's 16 MiB. Basic and Standard cap it at 8,388,608. The root's `DFE_KAFKA_MESSAGE_MAX_BYTES` output hands the dial's size to dfe-engine, which creates every topic at it. Redpanda Cloud's only creatable tier is Serverless, over a PrivateLink-style endpoint, and its SCRAM password reuses MSK's own seed. Hand-written Terraform for both still lives in `docs/deployment/kafka/confluent-cloud.md` and `redpanda-cloud.md`. - `strimzi` runs in the cluster exactly as it does anywhere else. Its broker autoscaling is a KEDA `ScaledObject` on the `KafkaNodePool` `/scale` subresource, triggered on consumer lag by default, with scale-in off until proven safe. ### ClickHouse - ClickHouse's storage model dial (`sizing.storage_model`) defaults to `auto`: object store by default wherever an endpoint exists, `local` is the explicit opt-out. On AWS that means `cached-object`: parts sit in the S3 bucket `terraform/modules/kubernetes-cluster/aws/object-store.tf` provisions, authenticated through its own Pod Identity role rather than a static key, behind a local read-through cache. The AWS ClickHouse shape pins a local-NVMe family (`r*d`, generation floor 8), and the resolver sizes a dedicated instance-store cache disk for it and points `clickhouse.objectStore.cache.volume` at `instance-store`, so the cache runs on that NVMe rather than sharing the data PVC -- see [storage.md](/v/2.2.1/dfe-infra/deployment/storage.md) for the mechanism and its default (`pvc`, bounded at 60% of the data PVC's space) for any shape with no such NVMe. - Both node paths assemble that NVMe as one RAID0 array under the kubelet: the Karpenter pool through `instanceStorePolicy`, the managed node group through a nodeadm NodeConfig its launch template carries. The cache takes 0.8 of it (`clickhouse.instance_store_cache_share`), and the bucket, the cache placement and the storage model all reach the chart on the cluster secret, so a fresh deploy repo needs no overlay for any of them. - `clickhouse.mode: external` connects to a server you brought, ClickHouse Cloud first, and deploys none. The dial's `clickhouse.host` travels as `endpoints.clickhouse_host`, and the root hands it, the ports, TLS and the user to the cluster secret as `dfe.hyperi.io/clickhouse_*`, which layer2-data, layer2-platform and layer2-apps turn into clickhouse-cluster's, network-policies', the otel collector's and every app's `clickhouse` values. The password is `TF_VAR_clickhouse_external_password`, never a dial field: the secrets module seeds it at `//clickhouse/external`, and clickhouse-cluster copies it into the engine's namespace as `clickhouse-admin-password`. `dfe-ops cloud-cycle` opens the service's IP access list to the cluster's NAT egress (`DFE_EGRESS_CIDRS`) and its runner after the apply and closes it at teardown (`scripts/clickhouse_cloud_access.py`). - Retention starts from a 24-hour assumed consumer downtime on the sized (`scale`) tier, plus any `archiver_lag_hours`; `single` and `slim` are not sized and keep the chart's fixed 72-hour profile. DLQ topics hold 7 days (168 hours) regardless of tier. ## Receiver ingress DFE pushes terabytes a day through the receiver, and every AWS front door bills by volume: a Network Load Balancer charges per LCU-hour (1 GB processed), a Classic ELB per GB, CloudFront and Global Accelerator per GB, WAF per request. The receiver's door is its own NLB, the cheapest of those per GB, and no protection with a volume or request charge sits on it. The default is `exposure.groups.ingest.door: loadbalancer` with an empty `allow` list, so a fresh deploy renders no NLB at all and the receiver stays on a ClusterIP. Naming the senders in the deploy repo's `exposure.groups.ingest.allow` renders the NLB (AWS Load Balancer Controller, IP targets, client address preserved, its frontend security group holding the list) on the Elastic IPs the AWS root allocates, one per public subnet. [INGEST-EDGE.md](/v/2.2.1/dfe-infra/INGEST-EDGE.md) describes the doors, and [edge.md](/v/2.2.1/dfe-infra/deployment/edge.md) carries the AWS tier table -- every door, its group, its tier, the dial key that turns it and its cost bucket. An NLB bills an hourly charge plus a per-LCU-hour charge, one LCU-hour being 1 GB processed for TCP/UDP (verified 2026-09-15 against AWS's Elastic Load Balancing pricing page). The hourly half is XS. The processing half is the transport cost of ingest on this door. AWS data transfer out still applies to the replies on every path, culvert included. The receiver NLB is created with one Elastic IP per public subnet. Those addresses exist, and AWS bills them, while the ingest door is closed and no NLB exists. `edge.aws.receiver_eips: false` turns them off. See [aws-operations.md](/v/2.2.1/dfe-infra/deployment/aws-operations.md) for node pools and images, admin UI exposure and upgrades, and [aws-teardown.md](/v/2.2.1/dfe-infra/deployment/aws-teardown.md) for teardown and lifecycle tags.