# Trusting the DFE certificate

> One of two CAs signs the gateway's `*.<domain>` certificate, chosen per deployment, and the modes ask different things of a developer box. One command says which is live, and the readiness gate and access summary print it too:

```bash
python3 scripts/dfe-ops ca --status --kubeconfig .tmp/dfe.kubeconfig
```

Release and deploy helpers around this one live in [deploy-helpers.md](https://github.com/hyperi-io/dfe-infra/blob/2.2.2/docs/deploy-helpers.md).

## Self-signed mode (the product default)

`tls.issuerName: dfe-internal-ca`. cert-manager mints a `CN=dfe-internal-ca` root (ECDSA P-384, ten years) and its ClusterIssuer signs the edge wildcard. There's no DNS-01 and no external credential, and it works on a split-horizon name public ACME can't validate. A browser warns, and the dfe-hyperdx iframe fails outright because it can't show the interstitial. Trust the root once:

```bash
python3 scripts/dfe-ops ca             # the PEM plus the install lines
python3 scripts/dfe-ops ca --install   # writes the file, does the non-root half
```

`--install` writes `.tmp/dfe-internal-ca.crt` and branches on `platform.system()`. On Linux it adds the certificate to the Chromium-family NSS store at `~/.pki/nssdb` (Brave and Chrome read that, not the system store, and it needs `libnss3-tools`) and prints `sudo cp <file> /usr/local/share/ca-certificates/` plus `sudo update-ca-certificates`. On macOS Chrome and Safari read the system keychain, so nothing runs without root and the one line printed is `sudo security add-trusted-cert -d -r trustRoot -k /Library/Keychains/System.keychain <file>`. Plain `dfe-ops ca` prints both.

With `tls.internalCA.persist.enabled` on, the root persists across a rebuild, so that trust is a one-off. The gateway chart pushes the minted root to the deployment's secret store once (`PushSecret`, `updatePolicy: IfNotExists`) and restores it before cert-manager can mint (`ExternalSecret`, `refreshPolicy: CreatedOnce`). bootstrap.sh renders that same pair ahead of Argo, so the restore is ordered before the Certificate rather than racing it. The root is reused and the leaf lifetimes rotate.

| Setting | Where | Effect |
|---|---|---|
| `tls.internalCA.persist.enabled` | gateway chart values | **Off by default.** An ExternalSecret against a store the deployment doesn't have is Degraded forever, which fails the Argo sync of a working deploy. Turn it on where there's a store. |
| `tls.internalCA.persist.secretStoreName` | gateway chart values | The store both halves use (default `dfe-secret-store`). Empty renders neither. |
| `DFE_CA_PERSIST` | bootstrap env | Whether bootstrap pre-applies the restore. Defaults on when `DFE_VAULT_SECRET_ID` is set. |
| `DFE_CA_RESTORE_TIMEOUT` | bootstrap env | Seconds to wait for the restore (default 60). A first bootstrap times out by design. |

### The store

The root is two entries, `<project>/<env>/pki/internal-ca-crt` (property `tls_crt`) and `<project>/<env>/pki/internal-ca-key` (property `tls_key`), because ESO checks `IfNotExists` per entry and the AWS provider checks the entry, not the property. The restore also reads `<project>/<env>/pki/internal-ca`, the single entry older releases wrote, and writes a complete pair from either layout or nothing. Properties are underscored because the Vault provider reads a property as a gjson path.

On AWS Secrets Manager the external-secrets role can create, write, tag and describe secrets under `<project>/<env>/pki/` and nowhere else (`terraform/modules/secrets/aws-sm`). ESO creates the root under the account's `aws/secretsmanager` key, not the deployment's own, so it outlives the key a teardown schedules for deletion. The CA secrets carry the cloud root's governance tag set, run tags included (`DFE_RESOURCE_TAGS`). Surviving a teardown holds for a persistent deployment. A tagged test run's root carries that run's tags, so the run's own sweep deletes it and the next run mints its own.

On a first bootstrap the restore ExternalSecret is `Ready=False` until the PushSecret stores the root. `CreatedOnce` then reads it once, writes it back, and stops. `bootstrap/smoke-test.sh` doesn't count a `CreatedOnce` ExternalSecret whose Secret exists.

## Estate-PKI mode (`tls.vault`)

In an estate that already runs a PKI, point the edge issuer at it and every client already trusts the certificate. There's nothing to install and no root to persist. Set it in the deploy repo's own overlay (`infra/envoy-gateway-config.yaml`), never in a tracked dfe-infra values file:

```yaml
tls:
  issuerName: dfe-estate-pki
  vault:
    server: https://vault.example.com:8200
    path: pki_tls/sign/<role>          # the sign role the AppRole policy grants
    caBundle: <base64 PEM chain that verifies the server's own TLS>
    appRole:
      roleId: <the cert-manager AppRole role_id>
```

Bootstrap seeds the AppRole SecretID from `DFE_CERTMANAGER_SECRET_ID` into Secret `cert-manager-approle` (key `secretId`), namespace `cert-manager`. The sign role must allow the deployment's wildcard and match `tls.privateKey` (ECDSA P-384 by default). `tls.acme.email` and `tls.vault.server` are exclusive, and the chart fails the render on both.

Three notes on the values:

- `issuerName` must NOT be `tls.internalCA.issuerName`. That name already belongs to the internal mesh CA's ClusterIssuer, and `dfe-ops ca --status` reads the wildcard's `issuerRef` against it to decide the mode.
- `caBundle` is the chain that verifies the PKI SERVER's own HTTPS, not the CA the PKI issues from. Take it off the handshake with `openssl s_client -connect <host>:<port> -showcerts`, keep every certificate after the leaf, and base64 the concatenation onto one line.
- Nothing needs deleting to flip an existing deployment. The gateway-shim owns the `dfe-wildcard-tls` Certificate, so changing `tls.issuerName` rewrites its `issuerRef` and cert-manager re-issues in place. The new leaf's `notBefore` is the proof it went round again.

Vault signs from the CSR's SANs, so the leaf carries an empty subject and a critical `subjectAltName`. That's correct, not a truncated certificate.

## In-cluster ClickHouse

A deployment that carries a CA runs its in-cluster hops over TLS with verification on. One without stays on plain HTTP. For ClickHouse that's one block, `clickhouse.tls` in `argocd/values/common.yaml`, which the server chart and every client chart read and judge by the same rule: `enabled` and a CA to sign with. Nothing is installed on a developer box, because only pods verify this certificate.

- With the edge module on (bootstrap's default) the certificate comes from `dfe-internal-ca`. The data and apps appsets set `clickhouse.tls.internalCA.present` from the cluster's `dfe.hyperi.io/edge` label, the same one that decides whether the gateway chart minting that issuer is deployed.
- `clickhouse.tls.issuerRef` names another cert-manager issuer (an estate PKI, AWS Private CA or corporate CA) in the deploy repo's `infra/common.yaml`, and turns TLS on with the edge module on or off. It has to sign for in-cluster Service names such as `dfe-clickhouse-clickhouse-headless.clickhouse.svc.cluster.local`. Public ACME can't, and a Vault sign role has to allow those names and their `*.` form.
- A cluster with the edge module off and no `issuerRef` stays on HTTP. No Certificate is requested, so nothing waits on one. dfe-docker Compose is HTTP the same way.
- clickhouse-cluster issues `dfe-clickhouse-tls` and serves HTTPS on 8443 and native TLS on 9440 beside 8123 and 9000. A renewal lands in the same Secret, which both modes mount as a directory.
- That Secret holds `tls.key`, so it's read only inside the clickhouse namespace, by a namespaced SecretStore that copies `ca.crt` into `dfe-clickhouse-tls-ca`. Only that CA-only Secret is copied into the app namespace, as `dfe-clickhouse-ca`, through a ClusterSecretStore whose conditions admit only the namespaces it copies into.
- dfe-engine reads the CA as `DFE_ENGINE_CLICKHOUSE_CA_CERT`, dfe-hyperdx as `NODE_EXTRA_CA_CERTS`, and dfe-loader through `SSL_CERT_FILE`, pointed at the image's own roots with the CA appended. That variable replaces the trust store rather than adding to it.
- An external ClickHouse sets `clickhouse.tls.port`, `verify` and `ca` for the server being dialled: `ca.configMapName` for a corporate CA bundle the deployment supplies, both `ca` names empty for a public certificate. dfe-loader can't skip verification and turns TLS on only for port 8443 or 9440 or a `clickhouse.cloud` host, so its chart refuses anything else.