DFE on AWS -- Operations

Provisioning (the dial, sizing, Kafka and ClickHouse choices) is in aws.md. Lifecycle tags and teardown are in aws-teardown.md. Stack, re-size and platform upgrades follow upgrades.md. upgrade-order.yaml at the repo root declares the chart upgrade order.

Kafka telemetry and autoscaling

• MSK's broker-count scaler is a Lambda that EventBridge runs every kafka.msk.autoscaling.period_seconds, wired from kafka.msk.autoscaling. Each run reads the cluster's current broker count and the BytesInPerSec of exactly those brokers over the last evaluation_periods. It adds step brokers when every period's per-broker mean was over per_broker_capacity_mb_s x headroom. The next run sees a broker added by this one, so pressure that outlasts a step brings another, up to max_brokers. It reads with the Average statistic, not Sum. BytesInPerSec is already a per-second rate, so summing the five one-minute samples in a 300s period would read roughly five times hot. The telemetry dial (telemetry.aws.sink, default otel) governs the rest. Broker logs land in an S3 bucket the fetcher reads, and only the opt-in cloudwatch sink keeps them in CloudWatch instead (the Telemetry section of docs/deployment/kafka/aws-msk.md has the cost table). open_monitoring is always scrape-only, read by the OTel collector gateway. Intelligent rebalancing is on, so a broker added later gets existing partitions moved onto it.
• Building the broker on its own: module.kafka has no module-wide depends_on, so tofu apply -target='module.kafka[0].aws_msk_cluster.this' plans the broker plus the VPC, its subnets and the KMS key and key policy, and nothing else. Targeting the whole module.kafka also pulls in the EKS control plane, because the bootstrap Job's Pod Identity association names the cluster and can't exist without it. The node groups, the addons and Karpenter stay out of both. -target and -exclude can't be combined ("The target and exclude planning options are mutually-exclusive"), so an MSK-only apply targets the module and excludes nothing.

Nodes and images

• The small always-on managed group (node_pools.system in the dial) carries only Karpenter and the AWS Load Balancer Controller, both at the same early sync wave. Each stateful workload the resolver sizes gets a fixed EKS managed node group at the resolved floor AND a node pool over the same instance family above it. Those workloads are kafka-broker, clickhouse, keeper, and kraft-controller where kafka.controller_pool is separate. The fixed group is what the workload keeps when Karpenter can't place a replacement. Every pool is arm64 and admits only the family the resolver picked, or the generations a cheapest-of policy compared, so a new Graviton generation arrives with the next resolve.
• Karpenter's node image is a dated amiAlias: al2023@v<YYYYMMDD>. scripts/resolve_sizing.py images writes the newest release that clears versions.yaml's node-images gates to sizing/node-pools.values.yaml. Commit that, with sizing/node-images.json, to the deploy repo's sizing/. Without it the chart default operators.karpenter-al2023-ami applies. Your own pools.aws.nodeClass.amiAlias in infra/node-pools.yaml beats both.

Image checks

• Every image a stack pins is checked for both linux/amd64 and linux/arm64 manifests before the stack ships, because an image missing either pulls fine and only fails at container start. scripts/check_image_arch.py --third-party-only --resolve-charts runs the same check with no registry credential.
• Every third-party image a pin tags deploys as tag@sha256, and the check reads it by that digest. Forgejo's curl helper comes from the chart values that pin it. Dex, MetalLB's FRR pair and Karpenter's digest-pinned controller come from their chart's own values at the pinned version. services-digests.karpenter and services.aws-load-balancer-controller-image move Karpenter's chart and the load balancer controller to digests too, once a stack pins them.
• Two pinned images still deploy by tag. One is the cluster-mode Redpanda broker, whose operator chart accepts only a bare semver tag. The other is Strimzi's broker, which its operator resolves.
• Some images are named in a chart render but never pulled, so we don't check them. Envoy Gateway's rate-limit image is one, because the gateway uses only rateLimit.local. cert-manager's ACME HTTP-01 solver is another, because every issuer here solves DNS-01. The last group is Strimzi's Kafka Bridge, Kafka Connect build and other-version Kafka images. No chart renders a KafkaBridge or KafkaConnect, and the Kafka resource names only services.kafka-version.
• One image we work around rather than ship. The third-party Cruise Control UI image is amd64-only, so DFE fetches its static release tarball at pod start and never runs that image.

Disruption budgets

Each Karpenter pool gets one budget per reason from its class, setting how many nodes may be terminating at once.

Pool	Underutilized	Empty	Drifted
kafka-broker, kraft-controller, clickhouse, keeper	0	1	1
eks-system, general	25%	25%	10%
ci-burst	100%	100%	100%

• k8s.disruption.maintenance_window (a UTC cron schedule, daily or one weekday, and a duration) holds drift to the window on every pool but ci-burst. Karpenter has no "only inside" budget, so the resolver writes a zero Drifted budget from the window's close to its next open.
• k8s.disruption.posture: conservative drops elastic Underutilized to 12% and replaces a stateful node for drift only inside the window. With no window, drift never replaces one.
• Expiry has no budget. The elastic pools' 720h expireAfter replaces a node whether the window is open or not.

Registry cache

Every node pulls its quay.io, registry.k8s.io and public.ecr.aws images through an ECR pull-through cache in the deployment's own account, which keeps the pulls off those registries' rate limits. None of the three needs a credential. Design: registry-cache/CONTRACT.md.

• It's on by default. registry_cache.enabled: "false" in the dial turns it off, and tofu output registry_cache lists what is cached.
• docker.io and ghcr.io images pull straight from upstream, anonymously, because no image the stack pulls needs a credential to read. A Docker Official Image comes from its public.ecr.aws/docker/library/ mirror instead of docker.io, so the cache covers it.
• A publisher's copy on one of the three cached registries wins over docker.io, so the default cache covers it. The Kafka CLI (kafka.cliImage) is Strimzi's Kafka image from quay.io. The single-mode broker still runs apache/kafka.
• Image references never change. Each node writes a containerd mirror per cached host with the upstream as fallback, so a cache that can't answer costs a retry. The mirror logs in with the node's own IAM role.
• Turning it on gives each managed node group its first launch template, which replaces the group.
• The deploying identity needs ECR's pull-through cache rule and repository creation template actions, iam:PassRole to ecr.amazonaws.com, and iam:CreateServiceLinkedRole the first time for both pullthroughcache.ecr.amazonaws.com and ecr.amazonaws.com (the AWSServiceRoleForECRTemplate role a repository creation template needs). Where the deploying identity may not create service-linked roles, create both once per account beforehand. A permissions boundary on the roles it creates must allow ecr:BatchImportUpstreamImage, ecr:CreateRepository, ecr:TagResource and kms:RetireGrant.
• code.forgejo.org and docker.redpanda.com have no ECR rule type. Argo CD and bootstrap fetch Helm charts, not nodes, so charts stay on their own registries.
• Destroy deletes every repository ECR created under the deployment's rules.

Optional: cache docker.io and ghcr.io too

This isn't part of the default deploy. ECR caches docker.io and ghcr.io only through a Secrets Manager credential, an AWS rule that holds whether the image is public or not. Where their rate limits bite, name one secret per upstream:

registry_cache:
  enabled: "true"
  credentials:
    docker-hub: "<secret ARN>"   # a Docker Hub username and personal access token
    ghcr: "<secret ARN>"         # a GitHub username and a classic token with read:packages

• Each secret is named under ecr-pullthroughcache/, holds {"username", "accessToken"}, and is encrypted with the default aws/secretsmanager key. ECR reads no other prefix and supports no customer managed key here.
• Leave either key out and that upstream gets no rule, so its images pull directly.
• Only ECR reads the secret, when it fetches from the upstream. No node or pod is handed it.

Admin UI exposure

edge.md owns which surface is exposed on which flavour, at which tier, by which dial key. Here's what's AWS's own:

• argocd/values/aws.yaml sets oidc.enabled: true. It stays with the cloud overlay and not the edge one because kafbat and dfe-engine read the same switch.
• The class-wide kill switch (edge.admin_uis.external) is off here, because this cloud's Envoy Gateway Service is internet-facing. The chart's render guard refuses to turn it back on for an internet-facing Service unless edge OIDC is on or an allow-list is set. So re-exposing an admin UI is a deliberate overlay change, not a one-line flip that ships with no edge auth.
• A public hostname needs dns.public_zone and gets a Let's Encrypt certificate by DNS-01, through cert-manager's own Pod Identity role.
• render_dial.py validates every boolean and enum in the dial's edge: block and reports which surfaces it marks public. The values that reach the chart are whatever a deploy-repo overlay pastes in from it.