# Trusting the DFE certificate > Which CA signed the gateway's `*.` certificate is a per-deployment > choice, and the two modes ask different things of a developer box. One command > says which is live, and the readiness gate and access summary print it too: ```bash python3 scripts/dfe-ops ca --status --kubeconfig .tmp/dfe.kubeconfig ``` Release and deploy helpers around this one live in [DEPLOY-HELPERS.md](https://github.com/hyperi-io/dfe-infra/blob/2.2.1/docs/DEPLOY-HELPERS.md). ## Self-signed mode (the product default) `tls.issuerName: dfe-internal-ca` -- cert-manager mints a `CN=dfe-internal-ca` root (ECDSA P-384, ten years) and its ClusterIssuer signs the edge wildcard. No DNS-01, no external credential, and it works on a split-horizon name public ACME cannot validate. The cost is trust: a browser warns, and the HyperDX iframe fails outright because it cannot show the interstitial. Trust the root once: ```bash python3 scripts/dfe-ops ca # the PEM plus the install lines python3 scripts/dfe-ops ca --install # writes the file, does the non-root half ``` `--install` writes `.tmp/dfe-internal-ca.crt` and branches on `platform.system()`. On Linux it adds the certificate to the Chromium-family NSS store at `~/.pki/nssdb` (Brave and Chrome read that, not the system store; needs `libnss3-tools`) and prints `sudo cp /usr/local/share/ca-certificates/` plus `sudo update-ca-certificates`. On macOS Chrome and Safari read the system keychain, so nothing runs without root and the one line printed is `sudo security add-trusted-cert -d -r trustRoot -k /Library/Keychains/System.keychain `. Plain `dfe-ops ca` prints both. **The root PERSISTS across a rebuild** (#238), so that trust is a one-off. The gateway chart pushes the minted root to the deployment's secret store once (`PushSecret`, `updatePolicy: IfNotExists`) and restores it before cert-manager can mint (`ExternalSecret`, `refreshPolicy: CreatedOnce`); bootstrap.sh renders that same pair ahead of Argo, so the restore is ordered before the Certificate rather than racing it. The root is reused; the leaf lifetimes rotate. | Setting | Where | Effect | |---|---|---| | `tls.internalCA.persist.enabled` | gateway chart values | **Off by default.** An ExternalSecret against a store the deployment does not have is Degraded forever, which fails the Argo sync of a working deploy. Turn it on where there is a store. | | `tls.internalCA.persist.secretStoreName` | gateway chart values | The store both halves use (default `dfe-secret-store`); empty renders neither. | | `DFE_CA_PERSIST` | bootstrap env | Whether bootstrap pre-applies the restore. Defaults on when `DFE_VAULT_SECRET_ID` is set. | | `DFE_CA_RESTORE_TIMEOUT` | bootstrap env | Seconds to wait for the restore (default 60). A first bootstrap times out by design. | ### The store The root is two entries, `//pki/internal-ca-crt` (property `tls_crt`) and `//pki/internal-ca-key` (property `tls_key`), because ESO checks `IfNotExists` per entry and the AWS provider checks the entry, not the property. The restore also reads `//pki/internal-ca`, the single entry older releases wrote, and writes a complete pair from either layout or nothing. Properties are underscored because the Vault provider reads a property as a gjson path. On AWS Secrets Manager the external-secrets role can create, write, tag and describe secrets under `//pki/` and nowhere else (`terraform/modules/secrets/aws-sm`). ESO creates the root under the account's `aws/secretsmanager` key, not the deployment's own, so it outlives the key a teardown schedules for deletion. The CA secrets carry the cloud root's governance tag set, run tags included (`DFE_RESOURCE_TAGS`). Surviving a teardown holds for a persistent deployment. A tagged test run's root carries that run's tags, so the run's own sweep deletes it and the next run mints its own. On a first bootstrap the restore ExternalSecret is `Ready=False` until the PushSecret stores the root; `CreatedOnce` then reads it once, writes it back, and stops. `bootstrap/smoke-test.sh` does not count a `CreatedOnce` ExternalSecret whose Secret exists. ## Estate-PKI mode (`tls.vault`) In an estate that already runs a PKI, point the edge issuer at it and every client trusts the certificate already -- nothing to install, no root to persist. Set it in the deploy repo's own overlay (`infra/envoy-gateway-config.yaml`), never in a tracked dfe-infra values file: ```yaml tls: issuerName: dfe-estate-pki vault: server: https://vault.example.com:8200 path: pki_tls/sign/ # the sign role the AppRole policy grants caBundle: appRole: roleId: ``` Bootstrap seeds the AppRole SecretID from `DFE_CERTMANAGER_SECRET_ID` into Secret `cert-manager-approle` (key `secretId`), namespace `cert-manager`. The sign role must allow the deployment's wildcard and match `tls.privateKey` (ECDSA P-384 by default). `tls.acme.email` and `tls.vault.server` are exclusive; the chart fails the render on both. Three things about the values that are easy to get wrong: - `issuerName` must NOT be `tls.internalCA.issuerName` -- that name already belongs to the internal mesh CA's ClusterIssuer, and `dfe-ops ca --status` reads the wildcard's `issuerRef` against it to decide the mode. - `caBundle` is the chain that verifies the PKI SERVER's own HTTPS, not the CA the PKI issues from. Take it off the handshake: `openssl s_client -connect : -showcerts`, keep every certificate after the leaf, base64 the concatenation onto one line. - Nothing needs deleting to flip an existing deployment. The gateway-shim owns the `dfe-wildcard-tls` Certificate, so changing `tls.issuerName` rewrites its `issuerRef` and cert-manager re-issues in place; the new leaf's `notBefore` is the proof it went round again. Vault signs from the CSR's SANs, so the leaf carries an empty subject and a critical `subjectAltName`. That is correct, not a truncated certificate. ## In-cluster ClickHouse A deployment that carries a CA runs its in-cluster hops over TLS with verification on; one without stays on plain HTTP. For ClickHouse that is one block, `clickhouse.tls` in `argocd/values/common.yaml`, which the server chart and every client chart read and judge by the same rule: `enabled` and a CA to sign with. Nothing is installed on a developer box: only pods verify this certificate. - With the edge module on (bootstrap's default) the certificate comes from `dfe-internal-ca`. The data and apps appsets set `clickhouse.tls.internalCA.present` from the cluster's `dfe.hyperi.io/edge` label, the same one that decides whether the gateway chart minting that issuer is deployed. - `clickhouse.tls.issuerRef` names another cert-manager issuer -- an estate PKI, AWS Private CA or corporate CA -- in the deploy repo's `infra/common.yaml`, and turns TLS on with the edge module on or off. It has to sign for in-cluster Service names such as `dfe-clickhouse-clickhouse-headless.clickhouse.svc.cluster.local`, so public ACME cannot, and a Vault sign role has to allow those names and their `*.` form. - A cluster with the edge module off and no `issuerRef` stays on HTTP: no Certificate is requested, so nothing waits on one. dfe-docker Compose is HTTP the same way. - clickhouse-cluster issues `dfe-clickhouse-tls` and serves HTTPS on 8443 and native TLS on 9440 beside 8123 and 9000. A renewal lands in the same Secret, which both modes mount as a directory. - That Secret holds `tls.key`, so it is read only inside the clickhouse namespace, by a namespaced SecretStore that copies `ca.crt` into `dfe-clickhouse-tls-ca`. Only that CA-only Secret is copied into the app namespace, as `dfe-clickhouse-ca`, through a ClusterSecretStore whose conditions admit only the namespaces it copies into. - dfe-engine reads the CA as `DFE_CLICKHOUSE_CA_CERT`, hyperdx as `NODE_EXTRA_CA_CERTS`, and dfe-loader through `SSL_CERT_FILE`, pointed at the image's own roots with the CA appended -- that variable replaces the trust store rather than adding to it. - An external ClickHouse sets `clickhouse.tls.port`, `verify` and `ca` for the server being dialled: `ca.configMapName` for a corporate CA bundle the deployment supplies, both `ca` names empty for a public certificate. dfe-loader cannot skip verification and turns TLS on only for port 8443 or 9440 or a `clickhouse.cloud` host, so its chart refuses anything else.