Skip to main content

DFE Observability Standard

One telemetry contract for the whole deployment: the dfe-infra platform services, dfe-engine, dfe-ui and the Rust apps (receiver, loader, fetcher, archiver, transforms). Every component reports to one OpenTelemetry Collector, the collector writes ClickHouse, and Observe reads those tables. Nothing pushes to Observe.

Where telemetry lands​

The collector's ClickHouse exporter writes otel_logs, otel_traces and the otel_metrics_* tables into the DFE database. Observe, the engine's app-management queries and its KEDA shim all read them there. The dfe-hyperdx image has no OTLP listener, so the collector's otlp/hyperdx exporter is off by default.

DeploymentCollectorDatabaseSet by
Kubernetes (dfe-infra)otel-collector gateway and daemonset, in the otel namespacedfeclickhouse.database in argocd/values/common.yaml
Docker (dfe-docker)the bundled otel-collector, on a profile that declares otel or with DFE_OTEL_ENABLED=truedfeDFE_OTEL_DATABASE

Both write dfe.otel_* by default, not dfe-hyperdx's default database. dfe-schemas declares the tables and dfe-engine applies them at its startup. The collector runs with create_schema: false, so it only inserts.

On Kubernetes the gateway also reads ClickHouse system tables, the Kubernetes API and the data operators' metrics endpoints. It re-exposes every metric on a Prometheus exporter for an external scraper.

Choosing the destination​

telemetry.mode in dfe-infra's argocd/values/common.yaml picks the OTLP endpoint every app exports to. telemetry.otlpEndpointTemplate in argocd/values/apps/_common.yaml resolves it into each app's OTEL_EXPORTER_OTLP_ENDPOINT. It recognises four values.

telemetry.modeApps export toOverride
hyperdx (default)the stack's collector, <project>-otel-collector-gateway.otel.svc.cluster.local:4317telemetry.hyperdxEndpoint, or telemetry.collectorNamespace for the namespace
receiverdfe-receiver.<release namespace>.svc.cluster.local:4317telemetry.receiverEndpoint
externala deployer's own OTLP backendtelemetry.externalEndpoint, empty by default
prometheusnothing, apps are scraped insteadnone

Any other value resolves to an empty endpoint, and an app with an empty endpoint exports nothing over OTLP. Nothing rejects an unknown mode, so a misspelt one fails silently. So does external with no externalEndpoint.

An endpoint with no scheme gets http://, or https:// when otel.tls is true. otel.endpoint overrides the resolved value for one deployment. It is exported as written, so it must carry its scheme, and scalo-service sets OTEL_EXPORTER_OTLP_PROTOCOL beside it (default grpc). dfe-ui and dfe-hyperdx export over OTLP/HTTP, so they take the resolved endpoint on port 4318.

In receiver mode, apps export to dfe-receiver's OTLP listener. It binds 4317 and 4318 only while the receiver's config.otlp.enabled is true, and with no telemetry.receiverEndpoint dfe-infra refuses to render the receiver until it is. Turning it on also serves OTLP on the receiver's load balancer. The receiver converts each record to JSON in the ClickHouse exporter's row shape (its default otlp.mode: hyperdx) and sends it down the normal data path. It lands wherever the receiver's routing and the loader's table mapping send it. No shipped rule sends it to the otel_* tables.

The mode moves only the apps' OTLP push. The collector runs in every mode, so container logs, scraped /metrics and its other inputs keep landing in the otel_* tables.

On Docker, DFE_OTEL_EXPORTER_ENDPOINT plays the same part. A profile that declares otel points it at the bundled collector, an endpoint of your own sends telemetry there instead, and empty exports nothing.

Logs​

Apps write structured JSON logs to stdout. The scalo logger is the reference for the output contract, masking of sensitive values included: scalo-py's docs/core-pillars/LOGGING.md and scalo-rs's docs/core-pillars/logging.md. A component in another language matches that output, not that code.

Stdout is the log path, so an app needs no OTLP logging SDK to reach otel_logs.

  • On Kubernetes, the collector daemonset's filelog receiver reads /var/log/pods on each node and forwards to the gateway. A record with no service.name takes the container name, then the workload, then the pod.
  • On Docker, the fluentd log driver sends each DFE container's stdout to the collector's fluentforward receiver on port 24224, and the container name becomes service.name. DFE_CONTAINER_LOGS_ENABLED=false puts every service back on json-file.

The infra provider's own logs are a second feed into the same otel tables, off the DFE ingest path. The cluster annotation dfe.hyperi.io/provider_logs_flavour picks the receiver, run in a collector of its own so a failed receiver restarts alone:

FlavourReceiver and source
none (default)nothing renders
awsaws_cloudwatch, polling the EKS audit and CloudTrail log groups
azureazure_event_hub, fed by a diagnostic setting
gcpgooglecloudpubsub, fed by a log sink into a Pub/Sub subscription
rancherfilelog on the control-plane nodes, reading the kube-apiserver audit log

Metrics, traces and identity​

Apps push metrics and traces over OTLP to the endpoint telemetry.mode resolves. In the Rust apps this comes from scalo-rs, whose metrics feature includes otel-metrics and otel-tracing.

Every DFE app also serves Prometheus /metrics, whatever its OTLP push is set to. scalo-service annotates each pod with prometheus.io/scrape, port and path unless telemetry.prometheus.scrape is false (scalo-rs charts/scalo-service/templates/_telemetry.tpl). On Kubernetes the gateway scrapes annotated pods labelled app.kubernetes.io/part-of: dfe while clusterTelemetry.dfeServices.enabled is on, which is the default.

KEDA scales on ScalingPressure, which the engine's KEDA shim reads from otel_metrics_gauge (src/dfe_engine/keda_shim/queries.yaml). The engine's metric names carry the dfe_ prefix from metric_prefix in src/dfe_engine/deployment_contract.py.

scalo-service sets OTEL_SERVICE_NAME on every app from otel.serviceName, else the release's full name. It becomes ServiceName in the tables. scalo-rs also tags deployment.environment.name from the app's environment setting.

The hunt runner pushes rather than being scraped​

The runner has no HTTP listener, so nothing scrapes it. It pushes on the OTLP endpoint the chart already wires and reports dfe_hunt_runs_total (hunt_id, outcome), dfe_hunt_run_duration_seconds, dfe_hunt_rows_written_total, dfe_hunt_overruns_total, dfe_hunt_claims_total (hunt_id, outcome), dfe_hunt_backlog, dfe_hunt_tick_duration_seconds, dfe_hunt_detections_capped_total (hunt_id, rule_id), dfe_hunt_detections_dropped_total (hunt_id, rule_id), dfe_hunt_clickhouse_unavailable_total (stage, reason), dfe_hunt_fire_failures_total (hunt_id, reason), dfe_hunt_fires_held_total (hunt_id), and the safe horizon and overlap metrics below. The backlog gauge runs the SAME due-count SQL the KEDA shim scales on, so a dashboard and the autoscaler cannot disagree about the queue. Each fire also logs one INFO line carrying the hunt, the rows it wrote and how long it took.

A ClickHouse the runner cannot use never ends the process. At startup an outage (reason=connection), a refused login (reason=authentication, the runner's user not made yet) or a full server (reason=capacity) is retried on the clickhouse.resilience back-off, jittered and with no budget, and a tick that hits either is skipped until the next poll. Each one adds 1 to dfe_hunt_clickhouse_unavailable_total with stage connect or tick, and the outage logs one WARN line when it starts.

A fire that fails holds its watermark, so the same fire stays due. The runner retries hunts.failure_backoff_initial_seconds (default 30, env DFE_ENGINE_HUNTS_FAILURE_BACKOFF_INITIAL_SECONDS) after the first failure, doubling to hunts.failure_backoff_max_seconds (default 600, env DFE_ENGINE_HUNTS_FAILURE_BACKOFF_MAX_SECONDS), jittered. Each failure adds 1 to dfe_hunt_fire_failures_total with reason set to the ClickHouse error category (not_found for a missing table) or the hunt's own refusal (empty_query, which one refused rule also gives, no_customers, unknown_customers), and each held fire adds 1 to dfe_hunt_fires_held_total. The failure logs one line when it starts, again when its cause changes and every hunts.failure_reminder_seconds (default 3600, env DFE_ENGINE_HUNTS_FAILURE_REMINDER_SECONDS) while it lasts, and one INFO line when the hunt runs again. A missing table logs at WARN, since the hunt runs once the table is created, and any other cause at ERROR. An operator's run-now goes through the wait once.

A rule compiled into a hunt writes at most hunts.max_detections_per_run detection rows per run (default 1000, env DFE_ENGINE_HUNTS_MAX_DETECTIONS_PER_RUN), cut to hunts.max_detections_per_run_ceiling (default 10000, env DFE_ENGINE_HUNTS_MAX_DETECTIONS_PER_RUN_CEILING). A rule that matches more writes the cap plus one summary row: nil matched_uuid, _json of {"dfe_capped": true, "matched": N, "written": cap, "rule_id": ...}, and an _org_id only when every match shares one. The run logs one WARN line with the counts, adds 1 to dfe_hunt_detections_capped_total and adds the unwritten matches to dfe_hunt_detections_dropped_total.

The watermark still advances and the run still records as completed, so the same flood is not re-run whole next fire. The next fire's overlap re-reads the end of it, and writes up to the cap again from the matches the cap left out there. A hunt that carries a direct query rather than rules runs uncapped, and says so in an INFO line each time its specs load.

A row whose _timestamp_load is inside a fire's window but which has not reached the replica the hunt reads is otherwise never matched. Two layers stop that, both on by default:

  • The safe horizon is the newest _timestamp_load every replica already holds, and a hunt never reads past it. The runner reads it before claiming a fire, as the server clock less the largest absolute_delay in system.replicas over the hunt's source and detection tables, less a 10 s commit allowance. A table with no delay never holds a fire. While the horizon is short of the fire, the fire stays due and unclaimed, and each poll adds 1 to dfe_hunt_safe_horizon_deferrals_total. Once it runs, dfe_hunt_safe_horizon_deferrals_per_fire records how many polls it waited. A hunt held for hunts.safe_horizon_max_deferral_seconds (default 300, cut to the hunt's interval, env DFE_ENGINE_HUNTS_SAFE_HORIZON_MAX_DEFERRAL_SECONDS), counted from the first poll it held across every fire since, runs anyway. It logs one WARN line, adds 1 to dfe_hunt_safe_horizon_cap_reached_total, and records the horizon it ran short of, so the next fire re-reads from there. hunts.safe_horizon (env DFE_ENGINE_HUNTS_SAFE_HORIZON) turns the layer off. dfe_hunt_safe_horizon_seconds_behind and dfe_hunt_safe_horizon_query_duration_seconds record each read. A read that fails adds 1 to dfe_hunt_safe_horizon_unreadable_total and holds the fire as unknown lag, timed on the runner's clock. If the cap lets it run, the next fire re-reads its whole window.
  • Each rule re-reads hunts.window_overlap_seconds before its watermark (default 120, at least 10, env DFE_ENGINE_HUNTS_WINDOW_OVERLAP_SECONDS) and leaves out every source row it already detected, keyed on hunt, rule and the matched row's _uuid. A row that becomes readable within the overlap plus the deferral cap is never missed. A later row is, and dfe_hunt_safe_horizon_cap_reached_total and dfe_hunt_safe_horizon_seconds_behind show the lag that reaches past that bound. The dedupe reads the answering replica's detections, so a duplicate detection and notification is possible only when the cap lets a fire run while that replica's detection table lags. dfe_hunt_safe_horizon_cap_ran_with_target_lag_total counts those fires. dfe_hunt_overlap_rows_reread_total counts the source rows re-read, dfe_hunt_duplicates_skipped_total the detections already in the read window, and dfe_hunt_detection_latency_seconds the time from a matched row's _timestamp_load to its detection's insert. A direct query cannot skip what it saw, so it reads no overlap.

dfe-ui​

dfe-ui registers OpenTelemetry through @vercel/otel once per server process when OTEL_EXPORTER_OTLP_ENDPOINT is set, with OTEL_SERVICE_NAME defaulting to dfe-ui (dfe-ui apps/dfe-core-ui/src/instrumentation.ts). An empty endpoint registers nothing, so prometheus mode starts no exporter. Its server output reaches otel_logs from stdout like any other pod.

Its server logger (dfe-ui apps/dfe-core-ui/src/core/server/logger/) writes one scalo-py-shaped JSON record per line to stdout (timestamp, level, target, message, fields), with scalo's field-name masking. LOG_LEVEL sets the threshold (default INFO), and LOG_FORMAT is json or text. Unset, it picks JSON in production or whenever an OTLP endpoint is set, and text on a dev terminal. NextAuth's logger and Next's onRequestError hook write through it, and request headers are never logged.

Acceptance tests​

Both deployments ship a self-monitoring check beside the core data-path test.

  • Kubernetes: dfe-infra's post-deploy smoke test (bootstrap/smoke-test-integration.sh, CORE 1) polls for up to 180s for an otel_logs row newer than 600s in the DFE database, by default. It proves the pipeline is streaming. It does not say which services report.
  • Docker: dfe-docker's make post asserts dfe.otel_* rows newer than five minutes, and an otel_logs row from each of dfe-engine, dfe-loader and dfe-receiver written since that container started.

References​