# Provider logs in dfe-hyperdx

> Self-monitoring has two feeds: the stack's own OTLP, and the infra provider's own logs (the EKS audit log, CloudTrail, a Rancher API server audit log). Both land in dfe-hyperdx.

Receivers in the otel collector read the provider's cloud log service, or the Rancher audit file, and send the rows through the collector gateway to the otel database dfe-hyperdx reads. They bypass DFE ingest: no fetcher source, no customer table.

Every row carries `cloud.provider` and `dfe.provider_log.source`, so dfe-hyperdx can filter on them. The rows are generic OTel logs, not typed columns.

## Flavours

`clusterTelemetry.providerLogs.flavour` defaults to `none`.

| Flavour | Source | Receiver | State |
|---|---|---|---|
| `aws` | EKS control-plane log group, CloudTrail's CloudWatch group | `aws_cloudwatch` (alpha) | Built. The cloud root sets it when `telemetry.sink` is `otel` and at least one of the two groups exists. |
| `rancher` | kube-apiserver audit log on the control-plane nodes | `filelog` | Built. The deployer sets the flavour. |
| `azure` | Diagnostic settings through an Event Hub | `azure_event_hub` | Chart only. No Azure cloud root exists. |
| `gcp` | A log sink through a Pub/Sub subscription | `googlecloudpubsub` | Chart only. No GCP cloud root exists. |

The receivers run in a workload of their own, `<project>-otel-collector-provider-logs`, with their own ServiceAccount. `aws_cloudwatch` makes an STS identity call and dereferences the result without checking it. A failed call exits the collector process. In its own pod that restarts the provider-logs container and leaves the gateway alone.

## AWS

The cloud root outputs the flavour and the two log group names. Bootstrap writes them to the cluster secret, `layer2-data.yaml` passes them to the chart, and the receiver reads the groups through a Pod Identity role scoped to those two.

Under the `otel` sink both groups keep `telemetry.aws.forced_retention_days` in the dial. That defaults to CloudWatch's shortest period and can be set to another it accepts. Under the `cloudwatch` sink the groups keep `retention_days` and nothing here reads them.

`telemetry.aws.eks_log_types` picks the EKS streams. It defaults to `audit`.

The role is created under the account's permissions boundary when the cloud guard configures one, and the boundary has to allow `logs:FilterLogEvents`. Without it the role's own grant has no effect, the receiver is denied on every poll, and no row arrives. The cloud guard's preflight refuses a run whose boundary lacks it.

The receiver keeps its read offsets on a small PersistentVolumeClaim, one replica, replaced with `Recreate`. A replacement pod resumes from the checkpoint. The lookback (`aws.initialLookback`) only applies when there is no checkpoint yet.

Recovery after an outage has three limits:
- The checkpoint. Events before it are already delivered.
- The CloudWatch retention. Events older than it are gone, whatever the checkpoint says.
- The exporter retry ceiling. If the gateway is down for longer than `maxElapsedTime`, the queued batches are dropped.

A missing group or a role without access is logged by the receiver on every poll and nothing more.

## Rancher

The audit log path, the node selector and the tolerations all come from values. The defaults are RKE2's: `/var/lib/rancher/rke2/server/logs/audit.log` and the `node-role.kubernetes.io/control-plane` label. Another distribution sets its own.

Turning API server auditing on is the operator's job. RKE2 writes that file only once an audit policy is set. With auditing off the file isn't there, the receiver finds nothing to read, and the collector stays up.

## Why this path

A dfe-fetcher source on the normal data path doesn't fit. dfe-fetcher reads only a static AWS key pair, so it can't use Pod Identity. It would also put self-monitoring on the managed ingest path, and self-monitoring has to keep reporting when ingest breaks.

The collector already ships a receiver per cloud, and its Go SDK picks up Pod Identity natively.

Under the `otel` sink, CloudWatch is a catch-up window and the otel database holds the history, so retention is pre-trimmed. Hence the shortest period by default.

What it costs. The AWS receivers are alpha, so we pin the collector and the cloud cycle tests it. The rows are generic OTel logs. GuardDuty and Security Hub findings have no receiver, so they stay out.

## What the cloud cycle checks

The integration smoke test requires a fresh EKS audit row from the AWS flavour. CloudTrail is reported and never fails the run, because management events are sparse on a quiet account.