Skip to main content

Day to day and troubleshooting

Start with the target you run.

TargetRun itFix it
Docker ComposeOperating the stack: exposure, resource limits, secrets, what survives make cleanTroubleshooting
KubernetesUpgrades, autoscalingThe troubleshooting toolbox: an opt-in shell inside the cluster's network
AWSDFE on AWS, operationsThe toolbox again, on an on-demand bastion

When events are accepted and nothing lands​

The ingest edge returns 2xx once it has accepted an event, which says nothing about ClickHouse. Events are accepted but nothing lands walks the order to check on Compose: the loader's readiness, the table, the topic, then the dead letters. The loader's own page covers its subcommands, metrics and dead-letter queue.

Where the logs are​

On Compose with self-monitoring on, docker compose logs dfe-loader fails with "configured logging driver does not support reading logs". The lines are in ClickHouse. Search the otel_logs source in the console, or query dfe.otel_logs by ServiceName. Kubernetes ships stdout to the same table through the collector's daemonset.

Passwords​

Reset the admin and breakglass passwords in the console, under Settings, Account Management, Security accounts. Users and OIDC says where the first logins are.