# How the stack reports on itself

> Three surfaces, three different questions.

| Surface | Answers | Where |
|---|---|---|
| Health endpoints | is this process alive, can it work right now | `/livez`, `/readyz` |
| Self-telemetry | what has the stack been doing | the `dfe.otel_*` tables |
| The self test | did an event actually get through | `make post` |

Ingested data is a different question -- [troubleshooting.md](/v/2.2.2/dfe-docker/troubleshooting.md).

## The health surface is three paths, with no aliases

Every DFE service serves `/livez`, `/readyz` and `/metrics` on its observability
port and nothing else. `/healthz` and the `/health/*` paths are retired and
**404 on every image pinned here**. `make check-compose` rejects any healthcheck
aimed at one.

| Service | Host port | Notes |
|---|---|---|
| dfe-receiver / loader / archiver | 9090, 9091, 9093 | all three paths |
| dfe-fetcher and transform instances | none | all three paths on container port 9090, over the Docker network |
| dfe-engine | 8003 | `/metrics` is on its container's own 9090, unpublished |
| dfe-ui | none | all three on `:3000`, alongside the app |
| dfe-proxy | 3000 | `/livez`, answered by Envoy itself |
| otel-collector | 13133 | the `health_check` extension |
| dfe-hyperdx | 8000 | `/health` -- upstream HyperDX's own convention |

These bind `DFE_BIND_HOST` (`127.0.0.1`), so curl them from the box, not your
laptop.

On dfe-ui, an unknown path returns the 200 app shell rather than a 404 -- Next.js
routes anything it does not recognise to the page. Only the three paths above are
real there.

## Compose picks liveness or readiness for you

Compose has no readiness condition. `depends_on: service_healthy` is the only
startup gate, so anything others wait on must report READINESS.

| Service | Path | Why |
|---|---|---|
| dfe-loader | `/readyz` | receiver and fetcher wait on it |
| dfe-engine | `/readyz` | loader and proxy wait on it, and it provisions their schema |
| dfe-proxy, dfe-hyperdx-proxy | Envoy admin `/ready` on 9901 | no curl in the image, so bash `/dev/tcp` sends the request |
| everything else | `/livez` | nothing gates on them |
| otel-collector | none | distroless, so nothing inside can probe `:13133` |

A loader gated on liveness reports healthy while unable to reach ClickHouse, and
the receiver starts against it.

Kubernetes has separate probes and ignores `HEALTHCHECK`, so it never chooses.

## Self-telemetry is pushed, never scraped

The stack's own telemetry leaves by a different door from the data it ingests, so
a broken ingest pipeline cannot take the reporting on it down too.

```mermaid
flowchart LR
    apps["DFE services"] -->|OTLP push :4317| col["otel-collector"]
    apps -->|stdout| drv["Docker log driver"]
    drv -->|fluent_forward :24224| col
    chsrv[(ClickHouse system tables)] -->|sql_query pull| col
    host("docker host /proc") -->|host_metrics pull| col
    col -->|self-telemetry :8888| col
    col -->|clickhouse exporter| ch[(dfe.otel_* tables)]
    ch -.->|queries it| hdx["dfe-hyperdx"]

    classDef on fill:#009E73,color:#ffffff,stroke:#000000
    class col,ch on
```

Services PUSH. dfe-hyperdx reads ClickHouse rather than receiving anything -- the
fork ships no OTLP receiver, so the collector's exporter writes the tables it
queries. This is the same chain Kubernetes runs.

Three of the collector's inputs are PULLS, not pushes, because their sources cannot
push. `host_metrics` reads the docker host's CPUs for the usage readings below.
`sql_query` reads ClickHouse's own `system.metrics`, `system.events` and
`system.parts` on a 30s interval -- the `:9363` Prometheus endpoint is not exposed
by the stack's image, and these system tables are readable on any provider,
managed ClickHouse included. The `prometheus` receiver scrapes the collector's own
`service.telemetry` endpoint on loopback, which is what puts queue depth and
refused/failed counts (`otelcol_*`) into the same pipeline as everything else.

`sql_query` and `prometheus` are what the pre-canned DFE ClickHouse Health and DFE Pipeline Health dashboards read. Without them those dashboards render empty here while working on Kubernetes.

`/metrics` is a SEPARATE pathway, for anything that scrapes. This repo ships
nothing that does. Enabling push does not disable it: a Prometheus estate and an
OTLP one are both served.

dfe-engine's Overview counts cores from the usage rollup dfe-schemas declares. That rollup reads the Kubernetes shape, so the collector reports Compose in it:

| Reading | Where it comes from here | Table |
|---|---|---|
| `k8s.container.cpu_request` | `sql_query/usage`: the CPU limit of each service the stack runs | `otel_metrics_gauge` |
| `k8s.pod.phase`, 2 (Running) | the same query, at the same instant, which the rollup requires beside each limit | `otel_metrics_gauge` |
| `system.cpu.logical.count` | `host_metrics`, through the host's `/proc` mounted read-only at `/hostfs/proc` | `otel_metrics_sum` |

All three carry `k8s.node.name`, the docker host's name, which joins a service's cores to its host's.

- Compose carries CPU limits where Kubernetes counts requests, so a service counts its ceiling. When limits add up to more than the host has, cores deployed reads higher than worker-node cores.
- A one-shot (`restart: "no"`) exits once its job is done and does not count. A service with no CPU limit (`DFE_SERVICE_CPUS=0`, say) counts every core on the host.
- `make dev`, `ci`, `up`, `infra` and `apply` read the limits from the compose model of what `up` starts and hand them to the collector as `DFE_USAGE_CPU_LIMITS` and `DFE_USAGE_NODE_NAME`. A collector started by a plain `docker compose up` has neither, so cores deployed reads as not measured. `make apply` recreates the collector when an added or deleted source moved them.

## Turning self-monitoring on and off

Every profile declares `otel: true`. `DFE_OTEL_ENABLED=false` turns it off.

| Variable | Default | Effect |
|---|---|---|
| `DFE_OTEL_ENABLED` | unset, the profile decides | starts the bundled collector |
| `DFE_OTEL_EXPORTER_ENDPOINT` | empty | where services push -- **empty exports nothing** |
| `DFE_OTEL_DATABASE` | `dfe` | database the collector writes |
| `DFE_OTEL_HEALTH_PORT` | `13133` | the `health_check` extension |
| `DFE_ENGINE_API_METRICS_BACKEND` in `env/engine.env` | `opentelemetry` | makes dfe-engine push. `prometheus` serves `/metrics` only |

The endpoint picks the shape. Enabling the profile points services at the bundled
collector. Name your own endpoint and they push there instead, which is how you
feed an external OTLP backend with no bundled collector at all.

The collector's OTLP ports are NOT published. Self-monitoring stays on the
Compose network. Note `4317`/`4318` on the host are dfe-receiver's OTLP INGEST
edge -- data coming in from your estate, pointing the other way.

## What reports

| Component | Pushes? | Why |
|---|---|---|
| the six Rust services | metrics and traces, when built against scalo >= 2.10.11 | scalo's `metrics` feature pulls `otel-metrics` and `otel-tracing`, so OTLP export is on by default (scalo-rs#30) |
| dfe-engine | metrics and traces, on the `opentelemetry` backend `env/engine.env` ships | scalo-py has an exporter. Its CLI defaults the backend to `opentelemetry` from v2.29.14 (scalo-py#11) |
| dfe-hunt-runner | metrics, as `dfe_hunt_*` | it has no listener to scrape and its default `opentelemetry` backend pushes on `OTEL_EXPORTER_OTLP_ENDPOINT` alone |
| dfe-ui | no metrics | `@opentelemetry/api` only, no SDK. Serves `/metrics` on `:3000` |
| every DFE service | LOGS, via Docker's log driver | nothing in the stack exports logs over OTLP, so stdout is shipped instead -- see below |

`OTEL_EXPORTER_OTLP_ENDPOINT` is set on every service regardless, so a rebuilt
component starts reporting with no config change here.

Each Rust service pushes the scalo chassis set -- the `worker_pool_*` family,
`process_*` and `container_*` -- plus whatever it defines itself; the loader adds
`rdkafka_*`. What decides whether a service reports is the scalo version its image
links, not this repo's configuration.

The `opentelemetry` backend is DUAL -- it pushes and keeps serving `/metrics`
(`readers=[otlp(grpc)->..., prometheus(/metrics)]`), so the switch costs a
scraping estate nothing.

## Container logs go through the log driver, not a file reader

Nothing in the stack exports logs over OTLP, so the lines a service writes to stdout reach ClickHouse a different way: Docker's own `fluentd` log driver sends them to the collector's `fluent_forward` receiver, which turns the driver's tag into `ServiceName` and writes `dfe.otel_logs`. Search them in dfe-hyperdx on the `otel_logs` source.

Kubernetes reads the same lines off the node with a DaemonSet filelog receiver.
That does not port: the files live under the daemon's data-root, which is not
`/var/lib/docker` on every host and is not on the host at all under Docker
Desktop, and listing that directory needs root. The log driver needs no host path
and no privilege.

**`docker compose logs` still reads these services.** Docker's dual logging keeps a local copy of each line, so the driver adds the collector without taking the CLI away. Only DFE's own services move onto the driver: ClickHouse, the broker, the proxies and dfe-hyperdx keep `json-file`. Put every service back with:

```
DFE_CONTAINER_LOGS_ENABLED=false
```

| Variable | Default | Effect |
|---|---|---|
| `DFE_CONTAINER_LOGS_ENABLED` | `true` | ships DFE container stdout when a collector runs |
| `DFE_CONTAINER_LOG_ADDRESS` | `tcp://<DFE_BIND_HOST>:<DFE_OTEL_FLUENT_PORT>` | where the DAEMON sends it |
| `DFE_OTEL_FLUENT_PORT` | `24224` | the collector's receiver, on `DFE_BIND_HOST` |

The driver runs with `fluentd-async`, so a container starts whether or not the
collector is up. Without it Docker refuses to create a container it cannot reach
a logging endpoint for, which would put the collector on the startup path of the
whole stack. Lines written while the collector is down queue in the daemon and
are sent when it answers.

It also runs with `fluentd-request-ack`, and the queue depends on it. The
collector's host port is Docker's port proxy, which accepts connections from the
moment the container starts, before the collector inside is listening, and drops
what it is sent. Without an ack the driver counts those lines as delivered, and a
service starting beside the collector loses its startup lines.

Of the host's own metrics only its CPUs are collected, for the usage readings.

## What a passing self test proves

`make post` makes up to seven claims and each one the profile can make must hold:

- **Ingest.** Three marked events posted at the profile's ingest edge come back
  as those exact rows in `dfe.main` inside 60s.
- **Self-monitoring**, when a collector is running. Rows in the `dfe.otel_*` tables
  NEWER than five minutes, and a row in `dfe.otel_logs` under each of
  `dfe-engine`, `dfe-loader` and `dfe-receiver` written since that container
  started. On the gRPC tier the loader logs at startup and is quiet after it, so
  a five-minute window can age its only lines out while the earlier claims run.
  Every service that pushes has fresh metric rows under its own name, and the hunt
  runner also reports `dfe_hunt_backlog` and `dfe_hunt_tick_duration_seconds`.
- **Observability**, when dfe-hyperdx is running. Its API returns the six seeded
  sources and at least one provisioned dashboard, read through the proxy that
  gives it an identity.
- **Console.** Those same marked rows read back through the engine query API.
- **Hunts.** A hunt created while the runner is up is picked up without a
  restart, and a run queued through the API detects exactly the events its rule
  matches and none of its near misses.
- **Subscription**, where the tier has a bus. The loader is fetching at least one
  topic. It skips on the gRPC tiers.
- **Routing**, where the engine renders the tier's app config. A source created
  through the API reaches the running receiver, and a record matching its rule
  lands in a table of its own. The source is named for the run (`postroute`
  and 8 hex digits) and deleted afterwards, then its table is dropped, because the engine's source delete leaves the table. A run first sweeps that shape's sources and tables older than an hour, which a killed run left.

Once every claim has run, the three ingest rows are deleted: a lightweight `DELETE`, or a mutation (`ALTER TABLE ... DELETE`) on a table whose projections refuse one. A delete that does not finish within 60s is reported, never failed.

The freshness window is what makes the second claim mean "streaming now" rather
than "streamed once". The first two are the pair the Kubernetes bootstrap smoke
asserts as CORE 1 and CORE 2.

`complete-single-node-stack` asserts them, plus the HTTP surface through the
proxy. Outcomes and opt-out:
[operating.md](/v/2.2.2/dfe-docker/operating.md#the-power-on-self-test-and-what-a-pass-proves).

## Related

- [operating.md](/v/2.2.2/dfe-docker/operating.md) -- exposure, limits, secrets, upgrades, the dial
- [troubleshooting.md](/v/2.2.2/dfe-docker/troubleshooting.md) -- reading stack state, diagnosing failures
- [architecture.md](/v/2.2.2/dfe-docker/architecture.md) -- components, transports, how data moves