# Configuration reference > Lookup, not narrative. Every variable the stack reads, what it does, and what it > defaults to. [.env.example](https://github.com/hyperi-io/dfe-docker/blob/406c01cb997337c517042dc1ea04cda78e45e77f/.env.example) is the annotated template these come > from. Image pins (`*_VERSION`) are written by `make stack VERSION=X.Y.Z` from the DFE stack SSoT. Compose hard-fails on an unset pin rather than resolving `latest`, so run it before any compose command -- [deploying.md](/v/2.2.1/dfe-docker/deploying.md). ## Environment Variables ### Profile Selection | Variable | Use | Default | |--------------------------------------------------|--------------------------------------------------------------------------------------|---------------------------------------------------------------------| | `DFE_PROFILE` | Override active profile from service_profiles.yaml | - | ### Host exposure Which interface each published port binds. Ports are grouped by audience -- [operating.md](/v/2.2.1/dfe-docker/operating.md#port-exposure-is-split-by-audience) has the per-port table and the UI classes. | Variable | Use | Default | |--------------------------------------------------|--------------------------------------------------------------------------------------|---------------------------------------------------------------------| | `DFE_INGRESS_BIND_HOST` | Interface the receiver ingest ports bind | `0.0.0.0` | | `DFE_BIND_HOST` | Interface the datastore, broker, metrics and internal gRPC ports bind | `127.0.0.1` | | `DFE_BIND_SCOPE` | Where every web UI publishes (`localhost`\|`all`); `make` resolves it to `DFE_UI_BIND_HOST`, and `all` requires `DFE_EXTERNAL_ORIGIN` | `localhost` | | `DFE_INFRA_UIS_EXTERNAL` | Kill switch -- `false` unpublishes every infra-class UI and beats their own flags | `true` | | `DFE_UI_EXTERNAL` | Publish the DFE UI (dfe-proxy `:3000`); product class, kill switch never covers it | `true` | | `DFE_ENGINE_API_EXTERNAL` | Publish the engine API (`:8003`); product class | `true` | | `DFE_KAFBAT_UI_EXTERNAL` | Publish Kafbat (`:8081`); infra class | `true` | | `DFE_HYPERDX_UI_EXTERNAL` | Publish HyperDX (`:8090` and its API `:8000`); infra class | `true` | ### External origin The scheme and host a browser reaches this box on: scheme and host only, no port and no trailing slash. Compose appends the port itself (`DFE_UI_PORT` for the console), so `make` refuses a value with a port, a path, a query, no scheme or a character no host holds, such as `;`. The same holds for `DFE_HYPERDX_APP_URL` and `DFE_OAUTH2_PROXY_EXTERNAL_ORIGIN`. Four surfaces build absolute URLs from the origin, and each is wrong when it says `localhost` and the browser did not: the DFE UI's post-logout redirect, the oauth2-proxy callbacks registered with the IdP, HyperDX's link-backs into the DFE UI, and the `frame-ancestors` policy the proxy serves HyperDX under. The first three send the browser somewhere it cannot follow; the fourth blocks the console's embedded views outright. It defaults to `http://localhost`, and nothing infers the real one -- so reaching the stack on a hostname without setting this is what sends every logout to `http://localhost:3000`. Set it on any deployment a browser reaches by anything other than `localhost`; `DFE_BIND_SCOPE=all` requires it, and `make` stops the start goals until it is set. With [console TLS](#console-tls-opt-in) on it must start with `https://`. The ancestors list keeps `http://localhost` and `http://127.0.0.1` at the UI port alongside it, so setting an origin never takes the console away from an operator working on the box itself. | Variable | Use | Default | |--------------------------------------------------|--------------------------------------------------------------------------------------|---------------------------------------------------------------------| | `DFE_EXTERNAL_ORIGIN` | Browser-facing scheme and host only, no port, for every absolute URL the stack builds | `http://localhost` | ### Console TLS (opt-in) `DFE_PROXY_TLS=true` serves the console (`DFE_UI_PORT`), the HyperDX embed (`DFE_HYPERDX_EMBED_PORT`) and both HyperDX origins (`DFE_HYPERDX_APP_PORT`, `DFE_HYPERDX_API_PORT`) over TLS 1.2 or 1.3 with one certificate. Key exchange prefers hybrid post-quantum X25519MLKEM768, then P-384; TLS 1.2 offers AEAD suites only, AES-256-GCM first. OIDC sign-in needs it on any hostname but `localhost`: outside a dev posture the engine marks its OIDC state and token cookies `Secure` and a browser drops a `Secure` cookie sent over plain http. `make certs` mints the pair into `DFE_PROXY_CERT_DIR`: a P-384 `console.crt` and `0600` `console.key` for the host `DFE_EXTERNAL_ORIGIN` names plus `localhost` and `127.0.0.1`, along with the `ca.crt` that signed them. It deletes the CA's private key once it has signed, so trusting `ca.crt` in the browser or OS trusts that one certificate and nothing else. It never replaces a file already there. To renew or change the hostname, delete the three files, run it again and trust the new `ca.crt`. Both proxies run as the user who runs `make` under TLS, so run it as that user. Each machine whose browser opens the console trusts `ca.crt` once, as a certificate authority. Copy `ca.crt` there and nothing else: `console.key` stays on the stack's host. The hostname must resolve on that machine too, through DNS or `/etc/hosts`. - Firefox keeps its own trust store: Settings, Privacy & Security, View Certificates, the Authorities tab, Import, then tick "Trust this CA to identify websites". - Chrome and Edge: open Manage certificates in Settings and import `ca.crt` as a trusted authority. The menu names differ between versions and platforms. - macOS, which Safari uses: `sudo security add-trusted-cert -d -r trustRoot -k /Library/Keychains/System.keychain ca.crt`. - Windows, from an administrator prompt: `certutil -addstore -f Root ca.crt`. - Debian and Ubuntu system tools such as `curl`: copy it to `/usr/local/share/ca-certificates/dfe-local-ca.crt`, then run `sudo update-ca-certificates`. A CA-issued certificate goes in the same two files instead, with the full chain in `console.crt`. Copy real files rather than linking them: a symlink, certbot's `live/` layout included, dangles inside the container, so `cp -L` the pair into place. Set `DFE_EXTERNAL_ORIGIN=https://dfe.example.com` and make `DFE_HYPERDX_APP_URL` an `https://` origin too if you set it. Register the engine's callback with the IdP as `https://:/api/v1/auth/oidc//callback`, without the port when `DFE_UI_PORT` is 443. `make` refuses to start the stack until both files are real files the make user can read, `openssl` accepts them as a matching certificate and unencrypted key, both origins start with `https://`, `DFE_NETWORK_SUBNET` overlaps no other Docker network and the `auth` profile is off. oauth2-proxy serves no TLS here, so its callbacks could not answer an `https://` origin. dfe-proxy's two ports serve TLS only. The HyperDX ports answer TLS and plain HTTP on the same port, because the engine's admin-link probe, `make post` and the oauth2-proxy upstreams dial them over http on the compose network. The engine builds the callback URL from `X-Forwarded-Proto`, which it believes from dfe-proxy alone. TLS therefore pins the compose network to `DFE_NETWORK_SUBNET`, gives dfe-proxy its last host address and allocates every other container from the first half, so no other container and not the bridge gateway can hold the address the engine trusts. The default `10.207.0.0/24` sits outside Docker's stock address pools and the fleet's `172.16.0.0/12` pool. Move it when a second stack on the same host also runs TLS. Move it too when `make` reports an overlap or the range clashes with a LAN or VPN route. It must be a private (RFC 1918) network from `/16` to `/24`. Turning `DFE_PROXY_TLS` on or off on a running stack needs `make down` first, because Docker cannot re-address a live network. `make up`, `make ci` and `make dev` stop the stack themselves; `make apply` and `make infra` refuse until you do. `make test-e2e` and `make test-resilience` run under the dial too, with the same prechecks as a start. Each test's stack chains the fragments `make ci` does, so the exposure opt-outs and container logs apply to it as well. The suite probes the console at `https://:` and verifies its certificate by chain and by name, never with verification off. A certificate from a public CA verifies against the system trust store. For a private CA, set `DFE_PROXY_CA_BUNDLE` to a PEM file holding the CA chain that signed `console.crt`, which after `make certs` is its `ca.crt`. Chrome takes no CA bundle, so the browser steps are pinned to the public key of the certificate that check has just verified. | Variable | Use | Default | |--------------------------------------------------|--------------------------------------------------------------------------------------|---------------------------------------------------------------------| | `DFE_PROXY_TLS` | Serve the console and HyperDX over TLS (`true`\|`false`) | `false` | | `DFE_PROXY_CERT_DIR` | Directory holding `console.crt` and `console.key`; a path starting `/`, `./` or `../` | `./certs` | | `DFE_NETWORK_SUBNET` | Compose network under TLS, private and `/16` to `/24`; dfe-proxy takes its last host | `10.207.0.0/24` | | `DFE_PROXY_CA_BUNDLE` | PEM CA chain the e2e suites verify the console certificate against | unset: the system trust store | ### Infra UI authentication (opt-in) Arms an oauth2-proxy per infra-UI origin -- [operating.md](/v/2.2.1/dfe-docker/operating.md#gating-the-infra-uis-with-oidc). Off by default; no other profile may require an issuer. | Variable | Use | Default | |--------------------------------------------------|--------------------------------------------------------------------------------------|---------------------------------------------------------------------| | `DFE_AUTH_ENABLED` | Arm the `auth` compose profile (also a `service_profiles.yaml` footprint key) | `false` | | `DFE_OIDC_ISSUER_URL` | OIDC issuer; required when the profile is armed | - | | `DFE_OIDC_CLIENT_ID` | OAuth client id; required when the profile is armed | - | | `DFE_OIDC_CLIENT_SECRET` | OAuth client secret; required when the profile is armed | - | | `DFE_OIDC_ALLOWED_GROUPS` | Groups allowed through the gate; blank means authn alone and is refused | `dfe-infra,dfe-admin` | | `DFE_OIDC_EMAIL_DOMAINS` | Extra email-domain restriction on top of the group check | `*` | | `DFE_OAUTH2_PROXY_COOKIE_SECRET` | Session cookie key, shared by all three proxies; `make init` generates 32 bytes | generated | | `DFE_OAUTH2_PROXY_COOKIE_DOMAIN` | Cookie domain; blank is a host-only cookie, which shares one session across the box | blank | | `DFE_OAUTH2_PROXY_COOKIE_SECURE` | Set the Secure cookie flag; `true` only behind TLS of your own, as the oauth2-proxies stay plain HTTP under console TLS | `false` | | `DFE_OAUTH2_PROXY_EXTERNAL_ORIGIN` | Scheme and host only, no port, the OAuth redirect URLs are built from; overrides `DFE_EXTERNAL_ORIGIN` | follows `DFE_EXTERNAL_ORIGIN` | | `DFE_OAUTH2_PROXY_VERSION` | Override the oauth2-proxy image pin; not SSoT-derived, keep the `tag@sha256` form | pinned in `docker-compose.yml` | ### Image Registry | Variable | Use | Default | |--------------------------------------------------|--------------------------------------------------------------------------------------|---------------------------------------------------------------------| | `DOCKER_DEFAULT_PLATFORM` | Image architecture to pull from docker (if not wanting automatic determination) | - | | `IMAGE_REGISTRY` | OCI registry hosting published `dfe-*` images | `ghcr.io/hyperi-io` | ### Dev Builds | Variable | Use | Default | |--------------------------------------------------|--------------------------------------------------------------------------------------|---------------------------------------------------------------------| | `DFE_SRC_ROOT` | Directory holding your local `dfe-*` checkouts; set = build (and `LIVE=1` mount, `LIVE_UI=1` run `next dev` over) those instead of the git cache | unset (git cache) | | `DFE_SRC_REMOTE` | Git base URL the managed cache clones component repos from | `https://github.com/hyperi-io` | | `DFE_SRC_REF` | Branch, tag or commit the managed cache builds | `main` | | `DFE_SRC_CACHE` | Location of the managed source cache | `~/.cache/dfe-docker/src` | ### DFE Components - General | Variable | Use | Default | |--------------------------------------------------|--------------------------------------------------------------------------------------|---------------------------------------------------------------------| | `LOG_LEVEL` | Log level (trace\|debug\|info\|warn\|error) | `info` | | `LOG_FORMAT` | Log format | `text` | ### DFE Archiver Only `single-kafka` runs dfe-archiver, so archiving needs a Kafka profile ([profiles.md](/v/2.2.1/dfe-docker/profiles.md)). | Variable | Use | Default | |--------------------------------------------------|--------------------------------------------------------------------------------------|---------------------------------------------------------------------| | `DFE_ARCHIVER_VERSION` | Version of dfe-archiver to use | none -- `make stack` pins it; unset is a hard-fail | | `DFE_ARCHIVER_PROMETHEUS_PORT` | Archiver Prometheus port | `9093` | ### DFE Fetcher | Variable | Use | Default | |--------------------------------------------------|--------------------------------------------------------------------------------------|---------------------------------------------------------------------| | `DFE_FETCHER_VERSION` | Version of dfe-fetcher to use | none -- `make stack` pins it; unset is a hard-fail | | `DFE_FETCHER_INGEST_PORT` | Fetcher ingest port | `8082` | | `DFE_FETCHER_PROMETHEUS_PORT` | Fetcher Prometheus port | `9094` | | `AWS_ACCESS_KEY_ID` | AWS access key. Goes in `env/fetcher.env`, NOT `.env` -- the fetcher config reads it via `env:` | - | | `AWS_SECRET_ACCESS_KEY` | AWS secret key. Same file as above | - | The AWS **region** is not an environment variable: it is a literal in each source's fetcher config (`region: us-west-2` in `resources/seed/sources/aws-*.yaml`), because only the two credential fields are `env:`-interpolated. ### DFE Loader | Variable | Use | Default | |--------------------------------------------------|--------------------------------------------------------------------------------------|---------------------------------------------------------------------| | `DFE_LOADER_VERSION` | Version of dfe-loader to use | none -- `make stack` pins it; unset is a hard-fail | | `DFE_LOADER_PROMETHEUS_PORT` | Loader Prometheus port | `9091` | | `DFE_LOADER_GRPC_PORT` | Host port for the loader's gRPC Push listener (container port 6000) | `50051` | ### DFE Receiver | Variable | Use | Default | |--------------------------------------------------|--------------------------------------------------------------------------------------|---------------------------------------------------------------------| | `DFE_RECEIVER_VERSION` | Version of dfe-receiver to use | none -- `make stack` pins it; unset is a hard-fail | | `DFE_RECEIVER_BEATS_PORT` | Receiver Beats port | `5044` | | `DFE_RECEIVER_GRPC_PORT` | Receiver Vector gRPC port, published once its line in `docker-compose.yml` is uncommented | `6000` | | `DFE_RECEIVER_HEC_PORT` | Receiver HEC port | `8088` | | `DFE_RECEIVER_HTTP_PORT` | Receiver HTTP port | `8080` | | `DFE_RECEIVER_OTLP_GRPC_PORT` | Receiver OTLP gRPC port | `4317` | | `DFE_RECEIVER_OTLP_HTTP_PORT` | Receiver OTLP HTTP port | `4318` | | `DFE_RECEIVER_PROMETHEUS_PORT` | Receiver Prometheus port | `9090` | ### DFE Transform Elastic | Variable | Use | Default | |--------------------------------------------------|--------------------------------------------------------------------------------------|---------------------------------------------------------------------| | `DFE_TRANSFORM_ELASTIC_VERSION` | Version of dfe-transform-elastic to use | none -- `make stack` pins it; unset is a hard-fail | | `DFE_TRANSFORM_ELASTIC_PROMETHEUS_PORT` | Transform Elastic Prometheus port | `9099` | ### DFE Transform Vector | Variable | Use | Default | |--------------------------------------------------|--------------------------------------------------------------------------------------|---------------------------------------------------------------------| | `DFE_TRANSFORM_VECTOR_VERSION` | Version of dfe-transform-vector to use | none -- `make stack` pins it; unset is a hard-fail | | `DFE_TRANSFORM_VECTOR_PROMETHEUS_PORT` | Transform Vector Prometheus port | `9095` | | `DFE_TRANSFORM_VECTOR_API_PORT` | Transform Vector API port | `8686` | ### DFE Transform VRL | Variable | Use | Default | |--------------------------------------------------|--------------------------------------------------------------------------------------|---------------------------------------------------------------------| | `DFE_TRANSFORM_VRL_VERSION` | Version of dfe-transform-vrl to use | none -- `make stack` pins it; unset is a hard-fail | | `DFE_TRANSFORM_VRL_PROMETHEUS_PORT` | Transform VRL Prometheus port | `9096` | ### ClickHouse | Variable | Use | Default | |--------------------------------------------------|--------------------------------------------------------------------------------------|---------------------------------------------------------------------| | `CLICKHOUSE_VERSION` | Version of ClickHouse to use | none -- `make stack` pins it from the DFE stack SSoT; unset is a hard-fail | | `CLICKHOUSE_HOST` | External ClickHouse host (skips Docker container) | `clickhouse` | | `CLICKHOUSE_HTTP_PORT` | Host publish of the bundled ClickHouse's HTTP port; services in the stack dial 8123 whatever it says, so it is safe to move | `8123` | | `CLICKHOUSE_NATIVE_PORT` | Host publish of the bundled ClickHouse's native port; services in the stack dial 9000 whatever it says, so it is safe to move | `9000` | | `CLICKHOUSE_EXTERNAL_HTTP_PORT` | HTTP port dfe-engine, dfe-hunt-runner, dfe-loader and the connection the engine hands HyperDX dial on an external `CLICKHOUSE_HOST`. Leave unset for the bundled container | `8123` | | `CLICKHOUSE_EXTERNAL_NATIVE_PORT` | Native port dfe-engine, dfe-hunt-runner and the collector dial on an external `CLICKHOUSE_HOST`; leave unset for the bundled container | `9000` | | `CLICKHOUSE_SECURE` | TLS to an external `CLICKHOUSE_HOST` for dfe-engine, dfe-hunt-runner and the connection the engine hands HyperDX (`https://`), on `true`, `1` or `yes`. The collector and the loader do not read it | `false` | | `CLICKHOUSE_DB` | ClickHouse initialisation database | `default` | | `CLICKHOUSE_USERNAME` | ClickHouse admin username, which only dfe-engine connects as | `default` | | `CLICKHOUSE_PASSWORD` | ClickHouse admin password, which only dfe-engine connects with | - | | `CLICKHOUSE_KEEPER_PASSWORD` | The bundled ClickHouse's embedded Keeper identity, which every znode is created under and which is fixed for the life of the `clickhouse-data` volume. Compose refuses to start without it | generated | | `DFE_HUNT_RUNNER_CLICKHOUSE_PASSWORD` | Password of `dfe_hunt_runner`, the hunt runner's own ClickHouse user; the engine creates the user on it and the runner connects with it | generated | | `DFE_LOADER_CLICKHOUSE_PASSWORD` | Password of `dfe_loader`, the loader's own ClickHouse user, which the engine creates on it | generated | | `DFE_QUERY_READER_CLICKHOUSE_PASSWORD` | Password of `dfe_query_reader`, the platform reader HyperDX's seeded connection uses, which the engine creates on it | generated | | `DFE_OTEL_WRITER_CLICKHOUSE_PASSWORD` | Password of `dfe_otel_writer`, the collector's own ClickHouse user, which the engine creates on it | generated | | `DFE_CLICKHOUSE_DEFAULT_TTL_DAYS` | Days every time-series table keeps rows, the OTel tables included; 0 disables the default TTL; a source or dfe-schemas TTL overrides it | `90` | ### Kafka - General | Variable | Use | Default | |--------------------------------------------------|--------------------------------------------------------------------------------------|---------------------------------------------------------------------| | `DFE_KAFKA_BOOTSTRAP_SERVERS` | The broker dfe-engine creates topics on and dfe-archiver, dfe-fetcher, dfe-loader, dfe-receiver, dfe-transform-elastic, dfe-transform-vector and dfe-transform-vrl read or write. Set it for an external broker | `kafka:9092`, the in-stack broker | | `KAFKA_BACKEND` | Kafka backend for `kafka` transport profiles | `redpanda` | | `KAFKA_PLAINTEXT_HOST_PORT` | Kafka plaintext host port (host-facing) | `19092` | | `KAFKA_PLAINTEXT_PORT` | Host publish of the broker's in-network listener; services in the stack dial `kafka:9092` whatever it says | `9092` | ### Apache Kafka | Variable | Use | Default | |--------------------------------------------------|--------------------------------------------------------------------------------------|---------------------------------------------------------------------| | `APACHE_KAFKA_VERSION` | Version of Apache Kafka to use | none -- `make stack` pins it from the DFE stack SSoT; unset is a hard-fail | | `KAFKA_ADVERTISED_LISTENERS` | Listener addresses advertised to clients/brokers | `PLAINTEXT://kafka:9092,PLAINTEXT_HOST://localhost:19092` | | `KAFKA_AUTO_CREATE_TOPICS_ENABLE` | Toggle auto creation of topics | `true` | | `KAFKA_CLUSTER_ID` | Name of the Kafka cluster | `dfe-docker-dev-cluster-01` | | `KAFKA_CONTROLLER_LISTENER_NAMES` | Listeners used by the controller | `CONTROLLER` | | `KAFKA_CONTROLLER_QUORUM_VOTERS` | Set of voters | `1@kafka:29092` | | `KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS` | Time (ms) group coordinator waits before initial rebalance | `0` | | `KAFKA_INTER_BROKER_LISTENER_NAME` | Listener used for communication between brokers | `PLAINTEXT` | | `KAFKA_LISTENER_SECURITY_PROTOCOL_MAP` | Map of listener names and security protocols | `CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT` | | `KAFKA_LISTENERS` | List of listeners | `PLAINTEXT://:9092,PLAINTEXT_HOST://:19092,CONTROLLER://:29092` | | `KAFKA_NODE_ID` | Node ID associated with the roles | `1` | | `KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR` | Replication factor for the offsets topic | `1` | | `KAFKA_PROCESS_ROLES` | Roles the process will use | `broker,controller` | | `KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR` | Replication factor for the transaction topic | `1` | | `KAFKA_TRANSACTION_STATE_LOG_MIN_ISR` | Minimum ISR for transaction topic | `1` | ### Kafka Redpanda | Variable | Use | Default | |--------------------------------------------------|--------------------------------------------------------------------------------------|---------------------------------------------------------------------| | `REDPANDA_VERSION` | Version of Redpanda to use | none -- `make stack` pins it from the DFE stack SSoT; unset is a hard-fail | | `REDPANDA_MEMORY` | Memory for the Redpanda broker (Seastar reserves this up front); `redpanda/start.sh` lowers it to what the container limit leaves after the host's `vm.min_free_kbytes` | `1G` | ### Kafka UI (Kafbat) | Variable | Use | Default | |--------------------------------------------------|--------------------------------------------------------------------------------------|---------------------------------------------------------------------| | `KAFBAT_ENABLED` | Toggle to turn on Kafbat | `true` | | `KAFBAT_VERSION` | Version of Kafbat to use | none -- `make stack` pins it from the DFE stack SSoT; unset is a hard-fail | | `KAFBAT_DYNAMIC_CONFIG_ENABLED` | Toggle runtime config changes | `true` | | `KAFBAT_KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS` | Kafka bootstrap server | `kafka:9092` | | `KAFBAT_KAFKA_CLUSTERS_0_NAME` | Kafka cluster name | `dfe-local` | | `KAFBAT_PORT` | Kafbat port | `8081` | ## Core DFE Components | Variable | Use | Default | |--------------------------------------------------|--------------------------------------------------------------------------------------|---------------------------------------------------------------------| | `DFE_CORE_ENABLED` | Toggle to turn on core components | `true` | | `DFE_ENGINE_VERSION` | Version of dfe-engine to use | none -- `make stack` pins it; unset is a hard-fail | | `DFE_ENGINE_PORT` | Port used by dfe-engine | `8003` | | `DFE_ENGINE_CONFIG_DIR` | Path to config directory | `/app/config` | | `DFE_ENGINE_SCHEMAS_DIR` | Path to schemas directory | `/app/schemas` | | `DFE_ENGINE_CONTENT_DIR` | Path the content volume mounts on, holding the emitted contracts and the seed library | `/app/content` | | `DFE_UI_VERSION` | Version of dfe-ui to use | none -- `make stack` pins it; unset is a hard-fail | | `DFE_UI_PORT` | Port used by dfe-ui | `3000` | | `DFE_UI_NODE_ENV` | Node environment of dfe-ui | `production` | ### App contracts, and the custom environment the console writes The console's app settings page reports what an app can be configured with, and it reads that from the app itself rather than from a list the engine carries. One `contract-` one-shot per app runs that app's pinned image, writes its config schema and capability catalogue into the shared content volume, and exits; the engine reads them at `DFE_APP_CONTRACT_DIR`. An app that cannot emit leaves its contract absent and the engine starts anyway, so a settings page with nothing on it for one app means that app's one-shot logged a failure -- `docker compose logs contract-`. A setting the app's own schema does not declare is written as an environment variable instead, into a file named for the Compose service: `env/dfe-loader.custom.env`, beside the `env/loader.env` that `make init` creates. Each app reads both, the custom file second, so a custom key beats the same key in the operator's file. The `environment:` block in `docker-compose.yml` beats both, which is what keeps a custom key from taking over the broker address or the warehouse credentials. Neither file has to exist. A per-source instance such as `dfe-transform-vrl-` reads its app's custom file and then its own, `env/dfe-transform-vrl-.custom.env`, so a key set on one source reaches that source's container alone. `make apply SERVICES=dfe-transform-vrl-` recreates it, the command the engine's response names. The engine alone writes `*.custom.env` and only for a service it renders. A key you set by hand goes in the operator's file. Earlier checkouts read `env/loader.custom.env` and its siblings, names the engine never wrote and nothing reads now: move any key you put there into `env/loader.env`. **A custom env write needs `make apply SERVICES=`, not a restart.** Compose reads `env_file` when it creates a container, so `docker compose restart` keeps the environment the container already had. A bare `docker compose up -d` misses what make resolves from the profile, the broker address included. The engine writes those files into the `env/` directory of the checkout it is started from: `DFE_DEPLOYMENT_APP_ENV_DIR` names it (`/app/app-env` inside the container, the sibling of `DFE_DEPLOYMENT_APP_CONFIG_DIR`) and it is a read-write bind mount of `./env`, so the deployment's own working tree is what changes when an operator saves a setting. The engine runs as its image's user, not yours, so the two meet on a group. `make init` (and every start target) makes `env/` group-writable and setgid, the engine container gets the checkout's group through `group_add` (`DFE_DEV_GID`, which the Makefile sets to `id -g`), and each custom file is written `0640` in that group. Compose run by anyone in that group reads it, and nobody else on the host does. A bare `docker compose` without make adds group `1000`. When the engine cannot give a file the directory's group, its log says so and names the fix. ## Release check Every DFE app asks `releases.hyperi.io` at startup whether a newer version exists. It sends one POST of the product, its version, the OS, the architecture and a one-way install id, and a failure costs a warning line. It is on by default, and two lines in `.env` change that for every app at once. | Variable | Use | Default | |--------------------------------------------------|--------------------------------------------------------------------------------------|---------------------------------------------------------------------| | `DFE_VERSION_CHECK_ENABLED` | `false` stops the check in every app, so none of them phones out | `true` | | `DFE_VERSION_CHECK_SEND_INSTANCE_ID` | `false` keeps the check and sends no install id | `true` | Compose maps them onto scalo's `version_check.enabled` and `version_check.send_instance_id` keys, in the env form scalo's cascade nests with. Each app reads those under its own prefix, so compose does the spelling: `DFE_LOADER_`, `DFE_RECEIVER_` and `DFE_FETCHER_` for those three, `ARCHIVER_` for the archiver (no `DFE_`), `DFE_TRANSFORM_` for vector and vrl, `DFE_TRANSFORM_ELASTIC_` for elastic, and no prefix for dfe-engine. dfe-ui and the hunt runner start no check. A compose `environment:` entry beats `env/.env`, so set the dial in `.env`. The third-party images keep their own telemetry settings, which this dial does not reach. ## Self-monitoring Every profile declares `otel: true`, which starts a collector that takes the stack's own telemetry over OTLP and writes the `dfe.otel_*` tables HyperDX reads. `DFE_OTEL_ENABLED=false` turns it off. The only scrape is the collector reading its own metrics on 127.0.0.1:8888 and its OTLP ports stay on the Compose network. Full picture, including which services report today: [observability.md](/v/2.2.1/dfe-docker/observability.md). | Variable | Use | Default | |--------------------------------------------------|--------------------------------------------------------------------------------------|---------------------------------------------------------------------| | `DFE_OTEL_ENABLED` | Toggle to start the collector | unset, the profile decides | | `DFE_OTEL_COLLECTOR_VERSION` | Version of otel/opentelemetry-collector-contrib | none -- `make stack` pins it; unset is a hard-fail | | `DFE_OTEL_EXPORTER_ENDPOINT` | Where services push; empty exports nothing | empty (the profile points it at the bundled collector) | | `DFE_OTEL_DATABASE` | ClickHouse database the collector writes | `dfe` | | `DFE_OTEL_HEALTH_PORT` | Collector `health_check` host port | `13133` | | `DFE_OTEL_COLLECTOR_LOG_LEVEL` | Collector's own log level | `warn` | | `DFE_OTEL_FLUENT_PORT` | Collector `fluent_forward` host port, where the daemon sends container stdout | `24224` | | `DFE_CONTAINER_LOGS_ENABLED` | Ship DFE container stdout; `false` puts every service back on `json-file` | `true` | | `DFE_CONTAINER_LOG_ADDRESS` | Where the DOCKER DAEMON sends it | `tcp://127.0.0.1:24224` | | `DFE_ENGINE_METRICS_BACKEND` | dfe-engine metrics backend; `opentelemetry` is dual (push + `/metrics`) | `prometheus` (the profile sets `opentelemetry`) | ## HyperDX Every profile declares `hyperdx: true` and `DFE_HYPERDX_ENABLED=false` turns it off. It starts `hyperdx` (API + App), the `dfe-hyperdx-proxy` that fronts both origins, a one-shot `dfe-dashboards` that copies the engine's shipped dashboards into a volume, plus `hyperdx-documentdb`, sharing the always-on ClickHouse. `hyperdx-documentdb` is PostgreSQL with the DocumentDB extensions and the Mongo-wire gateway HyperDX talks to, in one container, with its usage telemetry to a third party switched off. The gateway serves TLS only, on a certificate the one-shot `hyperdx-documentdb-tls` mints into a volume on first start and renews once it has 30 days left. HyperDX gets only the CA that signed it, verifies the gateway against it and logs in with SCRAM-SHA-256. HyperDX runs in `oidc-proxy` mode, as it does on Kubernetes: it verifies the engine's ES384 token against the engine's JWKS. A caller carrying none is unauthenticated. Signing in to the console is what signs a browser in here: dfe-ui mirrors the session into a `dfe_token` cookie and cookies ignore ports. The engine identifies with a service token of its own, which is how a source added through the console reaches every HyperDX team. That token's team, `DFE_HYPERDX_TEAM`, holds no connection and no sources, because HyperDX seeds a team only from a person's login, never from the service token. Every other team gets its connection and sources on its members' first login, from the engine's answer for that member: `main` and `hunts` everywhere, plus the otel and `clickhouse_system` sources on a platform team. Each team also gets the shipped dashboards those sources can render. From hyperi-io/dfe-hyperdx#106 on, a login the engine gives no connection is retried on a later request. The same toggle points the engine at HyperDX: with it on, the engine receives `DFE_HYPERDX_ENABLED=true` and `DFE_HYPERDX_BASE_URL=http://hyperdx:8000`, so its `/api/v1/hyperdx/*` surface answers instead of returning `503 hyperdx_absent`. | Variable | Use | Default | |--------------------------------------------------|--------------------------------------------------------------------------------------|---------------------------------------------------------------------| | `DFE_HYPERDX_ENABLED` | Toggle to start HyperDX + its proxy + DocumentDB. Ignored with `DFE_CORE_ENABLED=false`, since HyperDX signs callers in with the engine's token | unset, the profile decides | | `DFE_HYPERDX_VERSION` | Version of the dfe-hyperdx fork image to use | pinned by `make stack` from the SSoT (fail-loud, like every image) | | `DFE_HYPERDX_API_PORT` | HyperDX API host port, on `dfe-hyperdx-proxy` | `8000` | | `DFE_HYPERDX_APP_PORT` | HyperDX App UI host port, on `dfe-hyperdx-proxy` | `8090` | | `DFE_HYPERDX_APP_URL` | Scheme and host only, no port, the browser uses to reach HyperDX; overrides `DFE_EXTERNAL_ORIGIN` | follows `DFE_EXTERNAL_ORIGIN` | | `DFE_HYPERDX_BASE_URL` | Where the ENGINE reaches HyperDX; name an external one here | `http://hyperdx:8000` | | `DFE_HYPERDX_TEAM` | Team name when the token carries no group claim | `dfe` | | `DFE_HYPERDX_USAGE_STATS_ENABLED` | `true` sends upstream HyperDX's usage stats to `in-otel.hyperdx.io`, a third party | `false` | | `HYPERDX_THEME` | UI theme (NEXT_PUBLIC_THEME) | `dfe` | | `HYPERDX_POSTGRES_USER` | DocumentDB login HyperDX uses, a PostgreSQL role inside DocumentDB | `hyperdx` | | `HYPERDX_POSTGRES_PASSWORD` | Password for that login | `hyperdx` | | `HYPERDX_EXPRESS_SESSION_SECRET` | Signs HyperDX's session cookie (`EXPRESS_SESSION_SECRET`) | generated by `make init`; sentinel otherwise, which `make post` fails | | `HYPERDX_TOKEN_ENCRYPTION_KEY` | Encrypts the third-party tokens HyperDX stores (`TOKEN_ENCRYPTION_KEY`), 64 hex chars | generated by `make init`; empty (plain text) otherwise, which `make post` fails | | `HYPERDX_DOCUMENTDB_VERSION` | `documentdb-local` image version | none -- `make stack` pins it from the DFE stack SSoT, and unset is a hard-fail | ## Default Ports | Port | Service | Protocol | |-------|----------------------|--------------------| | 3000 | dfe-proxy | Web UI (envoy fronts dfe-ui, which publishes no host port) | | 8000 | hyperdx | API | | 8003 | dfe-engine | HTTP API | | 8080 | dfe-receiver | HTTP ingest | | 8081 | Kafbat | Web UI | | 8090 | hyperdx | App UI | | 8123 | ClickHouse | HTTP API | | 9000 | ClickHouse | Native protocol | | 9090 | dfe-receiver | Prometheus metrics | | 9091 | dfe-loader | Prometheus metrics | | 9092 | Kafka (any backend) | Plaintext | | 9093 | dfe-archiver | Prometheus metrics | | 13133 | otel-collector | health_check | | 19092 | Kafka (any backend) | Plaintext host | | 50051 | dfe-loader | gRPC Push, container port 6000 | Additional receiver ports (commented out by default in docker-compose.yml): 4317 (OTLP gRPC), 4318 (OTLP HTTP), 5044 (Beats), 6000 (Vector gRPC, which also needs `DFE_RECEIVER_GRPC__ENABLED=true`), 8088 (HEC). dfe-fetcher and the transforms run one instance per source. An instance publishes no host port. ## ClickHouse Schema **dfe-engine is the schema authority.** It ships `/app/schemas` inside its own image (pinned by `DFE_ENGINE_VERSION`) and creates the ClickHouse objects at startup; the loader pre-warms those schemas into its cache. Nothing in this repo provisions tables. - `dfe` - master database for all DFE related tables - `dfe.main` - catch-all for unrouted events, carrying `_tags` (JSON) among the profile columns That last detail matters more than it looks: the e2e suite and the power-on self test both identify their own rows by reading `_tags.marker` back out. If a deployment provisions a schema without `_tags`, both will correctly refuse to claim they verified anything. ## Container Images Images are published from the component repos: - `ghcr.io/hyperi-io/dfe-archiver` - `ghcr.io/hyperi-io/dfe-engine` - `ghcr.io/hyperi-io/dfe-fetcher` - `ghcr.io/hyperi-io/dfe-loader` - `ghcr.io/hyperi-io/dfe-receiver` - `ghcr.io/hyperi-io/dfe-transform-elastic` - `ghcr.io/hyperi-io/dfe-transform-vector` - `ghcr.io/hyperi-io/dfe-transform-vrl` - `ghcr.io/hyperi-io/dfe-ui` ## Related - [deploying.md](/v/2.2.1/dfe-docker/deploying.md) -- the dial, pinning, upgrades - [operating.md](/v/2.2.1/dfe-docker/operating.md) -- what these settings mean in production - [observability.md](/v/2.2.1/dfe-docker/observability.md) -- the self-monitoring variables in context