Trusting the DFE certificate

One of two CAs signs the gateway's *.<domain> certificate, chosen per deployment, and the modes ask different things of a developer box. One command says which is live, and the readiness gate and access summary print it too:

python3 scripts/dfe-ops ca --status --kubeconfig .tmp/dfe.kubeconfig

Release and deploy helpers around this one live in deploy-helpers.md.

Self-signed mode (the product default)

tls.issuerName: dfe-internal-ca. cert-manager mints a CN=dfe-internal-ca root (ECDSA P-384, ten years) and its ClusterIssuer signs the edge wildcard. There's no DNS-01 and no external credential, and it works on a split-horizon name public ACME can't validate. A browser warns, and the dfe-hyperdx iframe fails outright because it can't show the interstitial. Trust the root once:

python3 scripts/dfe-ops ca             # the PEM plus the install lines
python3 scripts/dfe-ops ca --install   # writes the file, does the non-root half

--install writes .tmp/dfe-internal-ca.crt and branches on platform.system(). On Linux it adds the certificate to the Chromium-family NSS store at ~/.pki/nssdb (Brave and Chrome read that, not the system store, and it needs libnss3-tools) and prints sudo cp <file> /usr/local/share/ca-certificates/ plus sudo update-ca-certificates. On macOS Chrome and Safari read the system keychain, so nothing runs without root and the one line printed is sudo security add-trusted-cert -d -r trustRoot -k /Library/Keychains/System.keychain <file>. Plain dfe-ops ca prints both.

With tls.internalCA.persist.enabled on, the root persists across a rebuild, so that trust is a one-off. The gateway chart pushes the minted root to the deployment's secret store once (PushSecret, updatePolicy: IfNotExists) and restores it before cert-manager can mint (ExternalSecret, refreshPolicy: CreatedOnce). bootstrap.sh renders that same pair ahead of Argo, so the restore is ordered before the Certificate rather than racing it. The root is reused and the leaf lifetimes rotate.

Setting	Where	Effect
tls.internalCA.persist.enabled	gateway chart values	Off by default. An ExternalSecret against a store the deployment doesn't have is Degraded forever, which fails the Argo sync of a working deploy. Turn it on where there's a store.
tls.internalCA.persist.secretStoreName	gateway chart values	The store both halves use (default dfe-secret-store). Empty renders neither.
DFE_CA_PERSIST	bootstrap env	Whether bootstrap pre-applies the restore. Defaults on when DFE_VAULT_SECRET_ID is set.
DFE_CA_RESTORE_TIMEOUT	bootstrap env	Seconds to wait for the restore (default 60). A first bootstrap times out by design.

The store

The root is two entries, <project>/<env>/pki/internal-ca-crt (property tls_crt) and <project>/<env>/pki/internal-ca-key (property tls_key), because ESO checks IfNotExists per entry and the AWS provider checks the entry, not the property. The restore also reads <project>/<env>/pki/internal-ca, the single entry older releases wrote, and writes a complete pair from either layout or nothing. Properties are underscored because the Vault provider reads a property as a gjson path.

On AWS Secrets Manager the external-secrets role can create, write, tag and describe secrets under <project>/<env>/pki/ and nowhere else (terraform/modules/secrets/aws-sm). ESO creates the root under the account's aws/secretsmanager key, not the deployment's own, so it outlives the key a teardown schedules for deletion. The CA secrets carry the cloud root's governance tag set, run tags included (DFE_RESOURCE_TAGS). Surviving a teardown holds for a persistent deployment. A tagged test run's root carries that run's tags, so the run's own sweep deletes it and the next run mints its own.

On a first bootstrap the restore ExternalSecret is Ready=False until the PushSecret stores the root. CreatedOnce then reads it once, writes it back, and stops. bootstrap/smoke-test.sh doesn't count a CreatedOnce ExternalSecret whose Secret exists.

Estate-PKI mode (tls.vault)

In an estate that already runs a PKI, point the edge issuer at it and every client already trusts the certificate. There's nothing to install and no root to persist. Set it in the deploy repo's own overlay (infra/envoy-gateway-config.yaml), never in a tracked dfe-infra values file:

tls:
  issuerName: dfe-estate-pki
  vault:
    server: https://vault.example.com:8200
    path: pki_tls/sign/<role>          # the sign role the AppRole policy grants
    caBundle: <base64 PEM chain that verifies the server's own TLS>
    appRole:
      roleId: <the cert-manager AppRole role_id>

Bootstrap seeds the AppRole SecretID from DFE_CERTMANAGER_SECRET_ID into Secret cert-manager-approle (key secretId), namespace cert-manager. The sign role must allow the deployment's wildcard and match tls.privateKey (ECDSA P-384 by default). tls.acme.email and tls.vault.server are exclusive, and the chart fails the render on both.

Three notes on the values:

• issuerName must NOT be tls.internalCA.issuerName. That name already belongs to the internal mesh CA's ClusterIssuer, and dfe-ops ca --status reads the wildcard's issuerRef against it to decide the mode.
• caBundle is the chain that verifies the PKI SERVER's own HTTPS, not the CA the PKI issues from. Take it off the handshake with openssl s_client -connect <host>:<port> -showcerts, keep every certificate after the leaf, and base64 the concatenation onto one line.
• Nothing needs deleting to flip an existing deployment. The gateway-shim owns the dfe-wildcard-tls Certificate, so changing tls.issuerName rewrites its issuerRef and cert-manager re-issues in place. The new leaf's notBefore is the proof it went round again.

Vault signs from the CSR's SANs, so the leaf carries an empty subject and a critical subjectAltName. That's correct, not a truncated certificate.

In-cluster ClickHouse

A deployment that carries a CA runs its in-cluster hops over TLS with verification on. One without stays on plain HTTP. For ClickHouse that's one block, clickhouse.tls in argocd/values/common.yaml, which the server chart and every client chart read and judge by the same rule: enabled and a CA to sign with. Nothing is installed on a developer box, because only pods verify this certificate.

• With the edge module on (bootstrap's default) the certificate comes from dfe-internal-ca. The data and apps appsets set clickhouse.tls.internalCA.present from the cluster's dfe.hyperi.io/edge label, the same one that decides whether the gateway chart minting that issuer is deployed.
• clickhouse.tls.issuerRef names another cert-manager issuer (an estate PKI, AWS Private CA or corporate CA) in the deploy repo's infra/common.yaml, and turns TLS on with the edge module on or off. It has to sign for in-cluster Service names such as dfe-clickhouse-clickhouse-headless.clickhouse.svc.cluster.local. Public ACME can't, and a Vault sign role has to allow those names and their *. form.
• A cluster with the edge module off and no issuerRef stays on HTTP. No Certificate is requested, so nothing waits on one. dfe-docker Compose is HTTP the same way.
• clickhouse-cluster issues dfe-clickhouse-tls and serves HTTPS on 8443 and native TLS on 9440 beside 8123 and 9000. A renewal lands in the same Secret, which both modes mount as a directory.
• That Secret holds tls.key, so it's read only inside the clickhouse namespace, by a namespaced SecretStore that copies ca.crt into dfe-clickhouse-tls-ca. Only that CA-only Secret is copied into the app namespace, as dfe-clickhouse-ca, through a ClusterSecretStore whose conditions admit only the namespaces it copies into.
• dfe-engine reads the CA as DFE_ENGINE_CLICKHOUSE_CA_CERT, dfe-hyperdx as NODE_EXTRA_CA_CERTS, and dfe-loader through SSL_CERT_FILE, pointed at the image's own roots with the CA appended. That variable replaces the trust store rather than adding to it.
• An external ClickHouse sets clickhouse.tls.port, verify and ca for the server being dialled: ca.configMapName for a corporate CA bundle the deployment supplies, both ca names empty for a public certificate. dfe-loader can't skip verification and turns TLS on only for port 8443 or 9440 or a clickhouse.cloud host, so its chart refuses anything else.