Skip to main content

Monitoring

Three questions, three places.

QuestionWhere
Is this process alive, and can it work right now?/livez, /readyz and /metrics on every DFE service. The console's search service keeps its own /health
What has the stack been doing?The dfe.otel_* tables in ClickHouse, read in the console
Did an event actually get through?make post on Docker. On Kubernetes, dfe-ops stack-deploy ends by sending an event through to ClickHouse and checking the stack's own telemetry landed

Where the telemetry goes​

The DFE apps push metrics and traces over OTLP to one OpenTelemetry Collector. Container logs reach the same collector from stdout. The collector writes otel_logs, otel_traces and the otel_metrics_* tables into the dfe database, and the console reads them there. Telemetry goes through the collector, not the ingest path, so a broken ingest pipeline doesn't stop the reporting on it.

Every app also serves Prometheus /metrics, whatever its OTLP push is set to. So it works with an existing Prometheus as well as the console.

To send telemetry somewhere else, set the destination:

  • Kubernetes: telemetry.mode in your deploy repo's infra/common.yaml takes hyperdx (the default), receiver, external or prometheus. external needs telemetry.externalEndpoint.
  • Docker: DFE_OTEL_EXPORTER_ENDPOINT. Empty exports nothing.

A misspelt telemetry.mode fails silently, because nothing rejects an unknown value. An app with an empty endpoint exports nothing.

Dashboards​

Eight ship with the engine: Overview, DFE Services, DFE Pipeline Health, DFE Throughput, DFE Kafka, and three on ClickHouse (Health, Internals, Overview). They arrive in the console read-only, provisioned within a minute. To change one, export it and import your own copy. A dashboard seeds only for a team that holds the sources it reads, so a tenant's team doesn't get the platform's.