Configuration reference

Lookup, not narrative. Every variable the stack reads, what it does, and what it defaults to. .env.example is the annotated template these come from.

Image pins (*_VERSION) are written by make stack VERSION=X.Y.Z from the DFE stack SSoT. Compose hard-fails on an unset pin rather than resolving latest, so run it before any compose command -- deploying.md.

Environment Variables

Profile Selection

Variable	Use	Default
DFE_PROFILE	Override active profile from service_profiles.yaml	-

Host exposure

Which interface each published port binds. Ports are grouped by audience -- operating.md has the per-port table and the UI classes.

Variable	Use	Default
DFE_INGRESS_BIND_HOST	Interface the receiver ingest ports bind	0.0.0.0
DFE_BIND_HOST	Interface the datastore, broker, metrics and internal gRPC ports bind	127.0.0.1
DFE_BIND_SCOPE	Where every web UI publishes (localhost|all); make resolves it to DFE_UI_BIND_HOST, and all requires DFE_EXTERNAL_ORIGIN	localhost
DFE_INFRA_UIS_EXTERNAL	Kill switch -- false unpublishes every infra-class UI and beats their own flags	true
DFE_UI_EXTERNAL	Publish the DFE UI (dfe-proxy :3000); product class, kill switch never covers it	true
DFE_ENGINE_API_EXTERNAL	Publish the engine API (:8003); product class	true
DFE_KAFBAT_UI_EXTERNAL	Publish Kafbat (:8081); infra class	true
DFE_HYPERDX_UI_EXTERNAL	Publish HyperDX (:8090 and its API :8000); infra class	true

External origin

The scheme and host a browser reaches this box on: scheme and host only, no port and no trailing slash. Compose appends the port itself (DFE_UI_PORT for the console), so make refuses a value with a port, a path, a query, no scheme or a character no host holds, such as ;. The same holds for DFE_HYPERDX_APP_URL and DFE_OAUTH2_PROXY_EXTERNAL_ORIGIN. Four surfaces build absolute URLs from the origin, and each is wrong when it says localhost and the browser did not: the DFE UI's post-logout redirect, the oauth2-proxy callbacks registered with the IdP, HyperDX's link-backs into the DFE UI, and the frame-ancestors policy the proxy serves HyperDX under. The first three send the browser somewhere it cannot follow; the fourth blocks the console's embedded views outright.

It defaults to http://localhost, and nothing infers the real one -- so reaching the stack on a hostname without setting this is what sends every logout to http://localhost:3000. Set it on any deployment a browser reaches by anything other than localhost; DFE_BIND_SCOPE=all requires it, and make stops the start goals until it is set.

With console TLS on it must start with https://.

The ancestors list keeps http://localhost and http://127.0.0.1 at the UI port alongside it, so setting an origin never takes the console away from an operator working on the box itself.

Variable	Use	Default
DFE_EXTERNAL_ORIGIN	Browser-facing scheme and host only, no port, for every absolute URL the stack builds	http://localhost

Console TLS (opt-in)

DFE_PROXY_TLS=true serves the console (DFE_UI_PORT), the HyperDX embed (DFE_HYPERDX_EMBED_PORT) and both HyperDX origins (DFE_HYPERDX_APP_PORT, DFE_HYPERDX_API_PORT) over TLS 1.2 or 1.3 with one certificate. Key exchange prefers hybrid post-quantum X25519MLKEM768, then P-384; TLS 1.2 offers AEAD suites only, AES-256-GCM first. OIDC sign-in needs it on any hostname but localhost: outside a dev posture the engine marks its OIDC state and token cookies Secure and a browser drops a Secure cookie sent over plain http.

make certs mints the pair into DFE_PROXY_CERT_DIR: a P-384 console.crt and 0600 console.key for the host DFE_EXTERNAL_ORIGIN names plus localhost and 127.0.0.1, along with the ca.crt that signed them. It deletes the CA's private key once it has signed, so trusting ca.crt in the browser or OS trusts that one certificate and nothing else. It never replaces a file already there. To renew or change the hostname, delete the three files, run it again and trust the new ca.crt. Both proxies run as the user who runs make under TLS, so run it as that user.

Each machine whose browser opens the console trusts ca.crt once, as a certificate authority. Copy ca.crt there and nothing else: console.key stays on the stack's host. The hostname must resolve on that machine too, through DNS or /etc/hosts.

• Firefox keeps its own trust store: Settings, Privacy & Security, View Certificates, the Authorities tab, Import, then tick "Trust this CA to identify websites".
• Chrome and Edge: open Manage certificates in Settings and import ca.crt as a trusted authority. The menu names differ between versions and platforms.
• macOS, which Safari uses: sudo security add-trusted-cert -d -r trustRoot -k /Library/Keychains/System.keychain ca.crt.
• Windows, from an administrator prompt: certutil -addstore -f Root ca.crt.
• Debian and Ubuntu system tools such as curl: copy it to /usr/local/share/ca-certificates/dfe-local-ca.crt, then run sudo update-ca-certificates.

A CA-issued certificate goes in the same two files instead, with the full chain in console.crt. Copy real files rather than linking them: a symlink, certbot's live/ layout included, dangles inside the container, so cp -L the pair into place.

Set DFE_EXTERNAL_ORIGIN=https://dfe.example.com and make DFE_HYPERDX_APP_URL an https:// origin too if you set it. Register the engine's callback with the IdP as https://<hostname>:<DFE_UI_PORT>/api/v1/auth/oidc/<provider>/callback, without the port when DFE_UI_PORT is 443.

make refuses to start the stack until both files are real files the make user can read, openssl accepts them as a matching certificate and unencrypted key, both origins start with https://, DFE_NETWORK_SUBNET overlaps no other Docker network and the auth profile is off. oauth2-proxy serves no TLS here, so its callbacks could not answer an https:// origin.

dfe-proxy's two ports serve TLS only. The HyperDX ports answer TLS and plain HTTP on the same port, because the engine's admin-link probe, make post and the oauth2-proxy upstreams dial them over http on the compose network.

The engine builds the callback URL from X-Forwarded-Proto, which it believes from dfe-proxy alone. TLS therefore pins the compose network to DFE_NETWORK_SUBNET, gives dfe-proxy its last host address and allocates every other container from the first half, so no other container and not the bridge gateway can hold the address the engine trusts. The default 10.207.0.0/24 sits outside Docker's stock address pools and the fleet's 172.16.0.0/12 pool. Move it when a second stack on the same host also runs TLS. Move it too when make reports an overlap or the range clashes with a LAN or VPN route. It must be a private (RFC 1918) network from /16 to /24.

Turning DFE_PROXY_TLS on or off on a running stack needs make down first, because Docker cannot re-address a live network. make up, make ci and make dev stop the stack themselves; make apply and make infra refuse until you do.

make test-e2e and make test-resilience run under the dial too, with the same prechecks as a start. Each test's stack chains the fragments make ci does, so the exposure opt-outs and container logs apply to it as well. The suite probes the console at https://<DFE_EXTERNAL_ORIGIN host>:<DFE_UI_PORT> and verifies its certificate by chain and by name, never with verification off.

A certificate from a public CA verifies against the system trust store. For a private CA, set DFE_PROXY_CA_BUNDLE to a PEM file holding the CA chain that signed console.crt, which after make certs is its ca.crt. Chrome takes no CA bundle, so the browser steps are pinned to the public key of the certificate that check has just verified.

Variable	Use	Default
DFE_PROXY_TLS	Serve the console and HyperDX over TLS (true|false)	false
DFE_PROXY_CERT_DIR	Directory holding console.crt and console.key; a path starting /, ./ or ../	./certs
DFE_NETWORK_SUBNET	Compose network under TLS, private and /16 to /24; dfe-proxy takes its last host	10.207.0.0/24
DFE_PROXY_CA_BUNDLE	PEM CA chain the e2e suites verify the console certificate against	unset: the system trust store

Infra UI authentication (opt-in)

Arms an oauth2-proxy per infra-UI origin -- operating.md. Off by default; no other profile may require an issuer.

Variable	Use	Default
DFE_AUTH_ENABLED	Arm the auth compose profile (also a service_profiles.yaml footprint key)	false
DFE_OIDC_ISSUER_URL	OIDC issuer; required when the profile is armed	-
DFE_OIDC_CLIENT_ID	OAuth client id; required when the profile is armed	-
DFE_OIDC_CLIENT_SECRET	OAuth client secret; required when the profile is armed	-
DFE_OIDC_ALLOWED_GROUPS	Groups allowed through the gate; blank means authn alone and is refused	dfe-infra,dfe-admin
DFE_OIDC_EMAIL_DOMAINS	Extra email-domain restriction on top of the group check	*
DFE_OAUTH2_PROXY_COOKIE_SECRET	Session cookie key, shared by all three proxies; make init generates 32 bytes	generated
DFE_OAUTH2_PROXY_COOKIE_DOMAIN	Cookie domain; blank is a host-only cookie, which shares one session across the box	blank
DFE_OAUTH2_PROXY_COOKIE_SECURE	Set the Secure cookie flag; true only behind TLS of your own, as the oauth2-proxies stay plain HTTP under console TLS	false
DFE_OAUTH2_PROXY_EXTERNAL_ORIGIN	Scheme and host only, no port, the OAuth redirect URLs are built from; overrides DFE_EXTERNAL_ORIGIN	follows DFE_EXTERNAL_ORIGIN
DFE_OAUTH2_PROXY_VERSION	Override the oauth2-proxy image pin; not SSoT-derived, keep the tag@sha256 form	pinned in docker-compose.yml

Image Registry

Variable	Use	Default
DOCKER_DEFAULT_PLATFORM	Image architecture to pull from docker (if not wanting automatic determination)	-
IMAGE_REGISTRY	OCI registry hosting published dfe-* images	ghcr.io/hyperi-io

Dev Builds

Variable	Use	Default
DFE_SRC_ROOT	Directory holding your local dfe-* checkouts; set = build (and LIVE=1 mount, LIVE_UI=1 run next dev over) those instead of the git cache	unset (git cache)
DFE_SRC_REMOTE	Git base URL the managed cache clones component repos from	https://github.com/hyperi-io
DFE_SRC_REF	Branch, tag or commit the managed cache builds	main
DFE_SRC_CACHE	Location of the managed source cache	~/.cache/dfe-docker/src

DFE Components - General

Variable	Use	Default
LOG_LEVEL	Log level (trace|debug|info|warn|error)	info
LOG_FORMAT	Log format	text

DFE Archiver

Only single-kafka runs dfe-archiver, so archiving needs a Kafka profile (profiles.md).

Variable	Use	Default
DFE_ARCHIVER_VERSION	Version of dfe-archiver to use	none -- make stack pins it; unset is a hard-fail
DFE_ARCHIVER_PROMETHEUS_PORT	Archiver Prometheus port	9093

DFE Fetcher

Variable	Use	Default
DFE_FETCHER_VERSION	Version of dfe-fetcher to use	none -- make stack pins it; unset is a hard-fail
DFE_FETCHER_INGEST_PORT	Fetcher ingest port	8082
DFE_FETCHER_PROMETHEUS_PORT	Fetcher Prometheus port	9094
AWS_ACCESS_KEY_ID	AWS access key. Goes in env/fetcher.env, NOT .env -- the fetcher config reads it via env:	-
AWS_SECRET_ACCESS_KEY	AWS secret key. Same file as above	-

The AWS region is not an environment variable: it is a literal in each source's fetcher config (region: us-west-2 in resources/seed/sources/aws-*.yaml), because only the two credential fields are env:-interpolated.

DFE Loader

Variable	Use	Default
DFE_LOADER_VERSION	Version of dfe-loader to use	none -- make stack pins it; unset is a hard-fail
DFE_LOADER_PROMETHEUS_PORT	Loader Prometheus port	9091
DFE_LOADER_GRPC_PORT	Host port for the loader's gRPC Push listener (container port 6000)	50051

DFE Receiver

Variable	Use	Default
DFE_RECEIVER_VERSION	Version of dfe-receiver to use	none -- make stack pins it; unset is a hard-fail
DFE_RECEIVER_BEATS_PORT	Receiver Beats port	5044
DFE_RECEIVER_GRPC_PORT	Receiver Vector gRPC port, published once its line in docker-compose.yml is uncommented	6000
DFE_RECEIVER_HEC_PORT	Receiver HEC port	8088
DFE_RECEIVER_HTTP_PORT	Receiver HTTP port	8080
DFE_RECEIVER_OTLP_GRPC_PORT	Receiver OTLP gRPC port	4317
DFE_RECEIVER_OTLP_HTTP_PORT	Receiver OTLP HTTP port	4318
DFE_RECEIVER_PROMETHEUS_PORT	Receiver Prometheus port	9090

DFE Transform Elastic

Variable	Use	Default
DFE_TRANSFORM_ELASTIC_VERSION	Version of dfe-transform-elastic to use	none -- make stack pins it; unset is a hard-fail
DFE_TRANSFORM_ELASTIC_PROMETHEUS_PORT	Transform Elastic Prometheus port	9099

DFE Transform Vector

Variable	Use	Default
DFE_TRANSFORM_VECTOR_VERSION	Version of dfe-transform-vector to use	none -- make stack pins it; unset is a hard-fail
DFE_TRANSFORM_VECTOR_PROMETHEUS_PORT	Transform Vector Prometheus port	9095
DFE_TRANSFORM_VECTOR_API_PORT	Transform Vector API port	8686

DFE Transform VRL

Variable	Use	Default
DFE_TRANSFORM_VRL_VERSION	Version of dfe-transform-vrl to use	none -- make stack pins it; unset is a hard-fail
DFE_TRANSFORM_VRL_PROMETHEUS_PORT	Transform VRL Prometheus port	9096

ClickHouse

Variable	Use	Default
CLICKHOUSE_VERSION	Version of ClickHouse to use	none -- make stack pins it from the DFE stack SSoT; unset is a hard-fail
CLICKHOUSE_HOST	External ClickHouse host (skips Docker container)	clickhouse
CLICKHOUSE_HTTP_PORT	Host publish of the bundled ClickHouse's HTTP port; services in the stack dial 8123 whatever it says, so it is safe to move	8123
CLICKHOUSE_NATIVE_PORT	Host publish of the bundled ClickHouse's native port; services in the stack dial 9000 whatever it says, so it is safe to move	9000
CLICKHOUSE_EXTERNAL_HTTP_PORT	HTTP port dfe-engine, dfe-hunt-runner, dfe-loader and the connection the engine hands HyperDX dial on an external CLICKHOUSE_HOST. Leave unset for the bundled container	8123
CLICKHOUSE_EXTERNAL_NATIVE_PORT	Native port dfe-engine, dfe-hunt-runner and the collector dial on an external CLICKHOUSE_HOST; leave unset for the bundled container	9000
CLICKHOUSE_SECURE	TLS to an external CLICKHOUSE_HOST for dfe-engine, dfe-hunt-runner and the connection the engine hands HyperDX (https://), on true, 1 or yes. The collector and the loader do not read it	false
CLICKHOUSE_DB	ClickHouse initialisation database	default
CLICKHOUSE_USERNAME	ClickHouse admin username, which only dfe-engine connects as	default
CLICKHOUSE_PASSWORD	ClickHouse admin password, which only dfe-engine connects with	-
CLICKHOUSE_KEEPER_PASSWORD	The bundled ClickHouse's embedded Keeper identity, which every znode is created under and which is fixed for the life of the clickhouse-data volume. Compose refuses to start without it	generated
DFE_HUNT_RUNNER_CLICKHOUSE_PASSWORD	Password of dfe_hunt_runner, the hunt runner's own ClickHouse user; the engine creates the user on it and the runner connects with it	generated
DFE_LOADER_CLICKHOUSE_PASSWORD	Password of dfe_loader, the loader's own ClickHouse user, which the engine creates on it	generated
DFE_QUERY_READER_CLICKHOUSE_PASSWORD	Password of dfe_query_reader, the platform reader HyperDX's seeded connection uses, which the engine creates on it	generated
DFE_OTEL_WRITER_CLICKHOUSE_PASSWORD	Password of dfe_otel_writer, the collector's own ClickHouse user, which the engine creates on it	generated
DFE_CLICKHOUSE_DEFAULT_TTL_DAYS	Days every time-series table keeps rows, the OTel tables included; 0 disables the default TTL; a source or dfe-schemas TTL overrides it	90

Kafka - General

Variable	Use	Default
DFE_KAFKA_BOOTSTRAP_SERVERS	The broker dfe-engine creates topics on and dfe-archiver, dfe-fetcher, dfe-loader, dfe-receiver, dfe-transform-elastic, dfe-transform-vector and dfe-transform-vrl read or write. Set it for an external broker	kafka:9092, the in-stack broker
KAFKA_BACKEND	Kafka backend for kafka transport profiles	redpanda
KAFKA_PLAINTEXT_HOST_PORT	Kafka plaintext host port (host-facing)	19092
KAFKA_PLAINTEXT_PORT	Host publish of the broker's in-network listener; services in the stack dial kafka:9092 whatever it says	9092

Apache Kafka

Variable	Use	Default
APACHE_KAFKA_VERSION	Version of Apache Kafka to use	none -- make stack pins it from the DFE stack SSoT; unset is a hard-fail
KAFKA_ADVERTISED_LISTENERS	Listener addresses advertised to clients/brokers	PLAINTEXT://kafka:9092,PLAINTEXT_HOST://localhost:19092
KAFKA_AUTO_CREATE_TOPICS_ENABLE	Toggle auto creation of topics	true
KAFKA_CLUSTER_ID	Name of the Kafka cluster	dfe-docker-dev-cluster-01
KAFKA_CONTROLLER_LISTENER_NAMES	Listeners used by the controller	CONTROLLER
KAFKA_CONTROLLER_QUORUM_VOTERS	Set of voters	1@kafka:29092
KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS	Time (ms) group coordinator waits before initial rebalance	0
KAFKA_INTER_BROKER_LISTENER_NAME	Listener used for communication between brokers	PLAINTEXT
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP	Map of listener names and security protocols	CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
KAFKA_LISTENERS	List of listeners	PLAINTEXT://:9092,PLAINTEXT_HOST://:19092,CONTROLLER://:29092
KAFKA_NODE_ID	Node ID associated with the roles	1
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR	Replication factor for the offsets topic	1
KAFKA_PROCESS_ROLES	Roles the process will use	broker,controller
KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR	Replication factor for the transaction topic	1
KAFKA_TRANSACTION_STATE_LOG_MIN_ISR	Minimum ISR for transaction topic	1

Kafka Redpanda

Variable	Use	Default
REDPANDA_VERSION	Version of Redpanda to use	none -- make stack pins it from the DFE stack SSoT; unset is a hard-fail
REDPANDA_MEMORY	Memory for the Redpanda broker (Seastar reserves this up front); redpanda/start.sh lowers it to what the container limit leaves after the host's vm.min_free_kbytes	1G

Kafka UI (Kafbat)

Variable	Use	Default
KAFBAT_ENABLED	Toggle to turn on Kafbat	true
KAFBAT_VERSION	Version of Kafbat to use	none -- make stack pins it from the DFE stack SSoT; unset is a hard-fail
KAFBAT_DYNAMIC_CONFIG_ENABLED	Toggle runtime config changes	true
KAFBAT_KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS	Kafka bootstrap server	kafka:9092
KAFBAT_KAFKA_CLUSTERS_0_NAME	Kafka cluster name	dfe-local
KAFBAT_PORT	Kafbat port	8081

Core DFE Components

Variable	Use	Default
DFE_CORE_ENABLED	Toggle to turn on core components	true
DFE_ENGINE_VERSION	Version of dfe-engine to use	none -- make stack pins it; unset is a hard-fail
DFE_ENGINE_PORT	Port used by dfe-engine	8003
DFE_ENGINE_CONFIG_DIR	Path to config directory	/app/config
DFE_ENGINE_SCHEMAS_DIR	Path to schemas directory	/app/schemas
DFE_ENGINE_CONTENT_DIR	Path the content volume mounts on, holding the emitted contracts and the seed library	/app/content
DFE_UI_VERSION	Version of dfe-ui to use	none -- make stack pins it; unset is a hard-fail
DFE_UI_PORT	Port used by dfe-ui	3000
DFE_UI_NODE_ENV	Node environment of dfe-ui	production

App contracts, and the custom environment the console writes

The console's app settings page reports what an app can be configured with, and it reads that from the app itself rather than from a list the engine carries. One contract-<app> one-shot per app runs that app's pinned image, writes its config schema and capability catalogue into the shared content volume, and exits; the engine reads them at DFE_APP_CONTRACT_DIR. An app that cannot emit leaves its contract absent and the engine starts anyway, so a settings page with nothing on it for one app means that app's one-shot logged a failure -- docker compose logs contract-<app>.

A setting the app's own schema does not declare is written as an environment variable instead, into a file named for the Compose service: env/dfe-loader.custom.env, beside the env/loader.env that make init creates. Each app reads both, the custom file second, so a custom key beats the same key in the operator's file. The environment: block in docker-compose.yml beats both, which is what keeps a custom key from taking over the broker address or the warehouse credentials. Neither file has to exist.

A per-source instance such as dfe-transform-vrl-<source> reads its app's custom file and then its own, env/dfe-transform-vrl-<source>.custom.env, so a key set on one source reaches that source's container alone. make apply SERVICES=dfe-transform-vrl-<source> recreates it, the command the engine's response names.

The engine alone writes *.custom.env and only for a service it renders. A key you set by hand goes in the operator's file. Earlier checkouts read env/loader.custom.env and its siblings, names the engine never wrote and nothing reads now: move any key you put there into env/loader.env.

A custom env write needs make apply SERVICES=<service>, not a restart. Compose reads env_file when it creates a container, so docker compose restart keeps the environment the container already had. A bare docker compose up -d misses what make resolves from the profile, the broker address included.

The engine writes those files into the env/ directory of the checkout it is started from: DFE_DEPLOYMENT_APP_ENV_DIR names it (/app/app-env inside the container, the sibling of DFE_DEPLOYMENT_APP_CONFIG_DIR) and it is a read-write bind mount of ./env, so the deployment's own working tree is what changes when an operator saves a setting.

The engine runs as its image's user, not yours, so the two meet on a group. make init (and every start target) makes env/ group-writable and setgid, the engine container gets the checkout's group through group_add (DFE_DEV_GID, which the Makefile sets to id -g), and each custom file is written 0640 in that group. Compose run by anyone in that group reads it, and nobody else on the host does. A bare docker compose without make adds group 1000. When the engine cannot give a file the directory's group, its log says so and names the fix.

Release check

Every DFE app asks releases.hyperi.io at startup whether a newer version exists. It sends one POST of the product, its version, the OS, the architecture and a one-way install id, and a failure costs a warning line. It is on by default, and two lines in .env change that for every app at once.

Variable	Use	Default
DFE_VERSION_CHECK_ENABLED	false stops the check in every app, so none of them phones out	true
DFE_VERSION_CHECK_SEND_INSTANCE_ID	false keeps the check and sends no install id	true

Compose maps them onto scalo's version_check.enabled and version_check.send_instance_id keys, in the env form scalo's cascade nests with. Each app reads those under its own prefix, so compose does the spelling: DFE_LOADER_, DFE_RECEIVER_ and DFE_FETCHER_ for those three, ARCHIVER_ for the archiver (no DFE_), DFE_TRANSFORM_ for vector and vrl, DFE_TRANSFORM_ELASTIC_ for elastic, and no prefix for dfe-engine. dfe-ui and the hunt runner start no check. A compose environment: entry beats env/<app>.env, so set the dial in .env. The third-party images keep their own telemetry settings, which this dial does not reach.

Self-monitoring

Every profile declares otel: true, which starts a collector that takes the stack's own telemetry over OTLP and writes the dfe.otel_* tables HyperDX reads. DFE_OTEL_ENABLED=false turns it off. The only scrape is the collector reading its own metrics on 127.0.0.1:8888 and its OTLP ports stay on the Compose network. Full picture, including which services report today: observability.md.

Variable	Use	Default
DFE_OTEL_ENABLED	Toggle to start the collector	unset, the profile decides
DFE_OTEL_COLLECTOR_VERSION	Version of otel/opentelemetry-collector-contrib	none -- make stack pins it; unset is a hard-fail
DFE_OTEL_EXPORTER_ENDPOINT	Where services push; empty exports nothing	empty (the profile points it at the bundled collector)
DFE_OTEL_DATABASE	ClickHouse database the collector writes	dfe
DFE_OTEL_HEALTH_PORT	Collector health_check host port	13133
DFE_OTEL_COLLECTOR_LOG_LEVEL	Collector's own log level	warn
DFE_OTEL_FLUENT_PORT	Collector fluent_forward host port, where the daemon sends container stdout	24224
DFE_CONTAINER_LOGS_ENABLED	Ship DFE container stdout; false puts every service back on json-file	true
DFE_CONTAINER_LOG_ADDRESS	Where the DOCKER DAEMON sends it	tcp://127.0.0.1:24224
DFE_ENGINE_METRICS_BACKEND	dfe-engine metrics backend; opentelemetry is dual (push + /metrics)	prometheus (the profile sets opentelemetry)

HyperDX

Every profile declares hyperdx: true and DFE_HYPERDX_ENABLED=false turns it off. It starts hyperdx (API + App), the dfe-hyperdx-proxy that fronts both origins, a one-shot dfe-dashboards that copies the engine's shipped dashboards into a volume, plus hyperdx-documentdb, sharing the always-on ClickHouse.

hyperdx-documentdb is PostgreSQL with the DocumentDB extensions and the Mongo-wire gateway HyperDX talks to, in one container, with its usage telemetry to a third party switched off. The gateway serves TLS only, on a certificate the one-shot hyperdx-documentdb-tls mints into a volume on first start and renews once it has 30 days left. HyperDX gets only the CA that signed it, verifies the gateway against it and logs in with SCRAM-SHA-256.

HyperDX runs in oidc-proxy mode, as it does on Kubernetes: it verifies the engine's ES384 token against the engine's JWKS. A caller carrying none is unauthenticated. Signing in to the console is what signs a browser in here: dfe-ui mirrors the session into a dfe_token cookie and cookies ignore ports. The engine identifies with a service token of its own, which is how a source added through the console reaches every HyperDX team. That token's team, DFE_HYPERDX_TEAM, holds no connection and no sources, because HyperDX seeds a team only from a person's login, never from the service token. Every other team gets its connection and sources on its members' first login, from the engine's answer for that member: main and hunts everywhere, plus the otel and clickhouse_system sources on a platform team. Each team also gets the shipped dashboards those sources can render. From hyperi-io/dfe-hyperdx#106 on, a login the engine gives no connection is retried on a later request.

The same toggle points the engine at HyperDX: with it on, the engine receives DFE_HYPERDX_ENABLED=true and DFE_HYPERDX_BASE_URL=http://hyperdx:8000, so its /api/v1/hyperdx/* surface answers instead of returning 503 hyperdx_absent.

Variable	Use	Default
DFE_HYPERDX_ENABLED	Toggle to start HyperDX + its proxy + DocumentDB. Ignored with DFE_CORE_ENABLED=false, since HyperDX signs callers in with the engine's token	unset, the profile decides
DFE_HYPERDX_VERSION	Version of the dfe-hyperdx fork image to use	pinned by make stack from the SSoT (fail-loud, like every image)
DFE_HYPERDX_API_PORT	HyperDX API host port, on dfe-hyperdx-proxy	8000
DFE_HYPERDX_APP_PORT	HyperDX App UI host port, on dfe-hyperdx-proxy	8090
DFE_HYPERDX_APP_URL	Scheme and host only, no port, the browser uses to reach HyperDX; overrides DFE_EXTERNAL_ORIGIN	follows DFE_EXTERNAL_ORIGIN
DFE_HYPERDX_BASE_URL	Where the ENGINE reaches HyperDX; name an external one here	http://hyperdx:8000
DFE_HYPERDX_TEAM	Team name when the token carries no group claim	dfe
DFE_HYPERDX_USAGE_STATS_ENABLED	true sends upstream HyperDX's usage stats to in-otel.hyperdx.io, a third party	false
HYPERDX_THEME	UI theme (NEXT_PUBLIC_THEME)	dfe
HYPERDX_POSTGRES_USER	DocumentDB login HyperDX uses, a PostgreSQL role inside DocumentDB	hyperdx
HYPERDX_POSTGRES_PASSWORD	Password for that login	hyperdx
HYPERDX_EXPRESS_SESSION_SECRET	Signs HyperDX's session cookie (EXPRESS_SESSION_SECRET)	generated by make init; sentinel otherwise, which make post fails
HYPERDX_TOKEN_ENCRYPTION_KEY	Encrypts the third-party tokens HyperDX stores (TOKEN_ENCRYPTION_KEY), 64 hex chars	generated by make init; empty (plain text) otherwise, which make post fails
HYPERDX_DOCUMENTDB_VERSION	documentdb-local image version	none -- make stack pins it from the DFE stack SSoT, and unset is a hard-fail

Default Ports

Port	Service	Protocol
3000	dfe-proxy	Web UI (envoy fronts dfe-ui, which publishes no host port)
8000	hyperdx	API
8003	dfe-engine	HTTP API
8080	dfe-receiver	HTTP ingest
8081	Kafbat	Web UI
8090	hyperdx	App UI
8123	ClickHouse	HTTP API
9000	ClickHouse	Native protocol
9090	dfe-receiver	Prometheus metrics
9091	dfe-loader	Prometheus metrics
9092	Kafka (any backend)	Plaintext
9093	dfe-archiver	Prometheus metrics
13133	otel-collector	health_check
19092	Kafka (any backend)	Plaintext host
50051	dfe-loader	gRPC Push, container port 6000

Additional receiver ports (commented out by default in docker-compose.yml): 4317 (OTLP gRPC), 4318 (OTLP HTTP), 5044 (Beats), 6000 (Vector gRPC, which also needs DFE_RECEIVER_GRPC__ENABLED=true), 8088 (HEC).

dfe-fetcher and the transforms run one instance per source. An instance publishes no host port.

ClickHouse Schema

dfe-engine is the schema authority. It ships /app/schemas inside its own image (pinned by DFE_ENGINE_VERSION) and creates the ClickHouse objects at startup; the loader pre-warms those schemas into its cache. Nothing in this repo provisions tables.

• dfe - master database for all DFE related tables
• dfe.main - catch-all for unrouted events, carrying _tags (JSON) among the profile columns

That last detail matters more than it looks: the e2e suite and the power-on self test both identify their own rows by reading _tags.marker back out. If a deployment provisions a schema without _tags, both will correctly refuse to claim they verified anything.

Container Images

Images are published from the component repos:

• ghcr.io/hyperi-io/dfe-archiver
• ghcr.io/hyperi-io/dfe-engine
• ghcr.io/hyperi-io/dfe-fetcher
• ghcr.io/hyperi-io/dfe-loader
• ghcr.io/hyperi-io/dfe-receiver
• ghcr.io/hyperi-io/dfe-transform-elastic
• ghcr.io/hyperi-io/dfe-transform-vector
• ghcr.io/hyperi-io/dfe-transform-vrl
• ghcr.io/hyperi-io/dfe-ui

Related

• deploying.md -- the dial, pinning, upgrades
• operating.md -- what these settings mean in production
• observability.md -- the self-monitoring variables in context