# Troubleshooting toolbox

> Sometimes an operator needs a shell inside the cluster's own network -- to
> curl a Service by its internal DNS name, run a query against ClickHouse
> without a port-forward, or poke at DNS resolution from where a DFE pod
> actually sits. Two surfaces share one image for this: an opt-in in-cluster
> pod, and an on-demand EC2 bastion for AWS deployments. Both are off by
> default.

## The image

`docker/dfe-toolbox/` builds four images (`linux/amd64` and `linux/arm64`), pushed by `.github/workflows/toolbox-build.yml` to `ghcr.io/hyperi-io/dfe-toolbox-{base,aws,gcp,azure}`. All four are Debian `trixie-slim`, pinned by digest (`toolbox.debian`), run as uid/gid 1000, and share one entrypoint. gcloud and az each add close to 1 GB, so they stay out of the base -- `-aws`, `-gcp` and `-azure` each `FROM` the base image and add one cloud's CLI on top.

**Base:** kubectl, helm, argocd (its CLI pinned to the ArgoCD *app* version
`bootstrap.argocd`'s chart deploys, so it stays inside its supported skew),
tofu, clickhouse-client (pinned to `services.clickhouse-version`, never a pin
of its own), psql, kcat (covers Kafka and Redpanda -- Apache's own console
scripts need a full Kafka install plus a JVM), and jq/yq/openssl/curl/dig/nc.
No `tcpdump`: the pod is unprivileged with no `CAP_NET_RAW`/`CAP_NET_ADMIN`,
so the binary would be dead weight.

Every version is a Docker build ARG defaulting to a `versions.yaml` `toolbox.*` pin. `scripts/check_versions_drift.py` holds each Dockerfile's default to its pin, and the build workflow overrides every one from the same source. A published family tag (`toolbox.dfe-toolbox`) never moves: a change to any image's inputs ships under a bumped tag, and the pull-request check fails until the bump is there.

Every download is checked before it installs, and a failed check fails the build:

- kubectl, tofu, clickhouse-client, gcloud and az come from the vendor's signed apt repo, with the repo key pinned by fingerprint in the Dockerfile and trusted for that repo only.
- argocd and yq are copied out of their official images, pinned by index digest (`toolbox.argocd-cli-image`, `toolbox.yq-image`). The build fails unless each binary reports its release pin.
- helm, the AWS CLI and the Session Manager plugin are checked against a signature from a key pinned by fingerprint. AWS publishes its two keys only in its docs, so they sit beside the aws Dockerfile.

**Per-cloud:** `-aws` adds the AWS CLI v2 and the Session Manager plugin, both
at the pinned version. `-gcp` adds gcloud. `-azure` adds az from Microsoft's apt repo pinned to `bookworm`
(`packages.microsoft.com` has no `trixie` suite yet; the build itself is a
self-contained venv that runs on trixie regardless).

**Writable paths:** `$HOME` (`/home/toolbox`) and `/tmp` only -- kubeconfig
cache, helm's cache, `~/.aws`, `~/.config/gcloud` and `~/.azure` all land
under `$HOME`. The root filesystem is read-only wherever this image runs. The
in-cluster pod's own `/work` mount, below, covers neither path.

## Bastion on AWS

If the in-cluster pod (below) cannot help -- the fault IS the Kubernetes API,
or the API and brokers are only reachable from inside the VPC -- the AWS
bastion is a standing-nothing alternative: an EC2 instance that exists only
while someone is using it, reached over Systems Manager Session Manager,
never SSH.
`terraform/modules/toolbox/aws` is the module; `dfe-ops bastion` is the
operator surface; `CONTRACT.md` there holds the exhaustive detail.

The shape resolver's `toolbox` shape (the newest Graviton generation in the
region), AL2023, private subnet, no public IP, no inbound security-group rule at
all. A different type goes in `sizing.overrides.toolbox.instance_type`, never in
the dial. User-data installs the same tool list the
image carries, at the same pins. The instance is created on `bastion up` and
TERMINATED (never stopped) on `bastion down`; a systemd timer on the instance
counts active SSM sessions and self-terminates after `toolbox.ttl_minutes`
(15-480, default 60) with none.

```yaml
toolbox:
  enabled: "false"          # dfe-ops bastion up/down flips this
  aws:
    shape_ref: toolbox      # a resolved shape, never an instance type
    operator_role_arn: ""   # required once enabled -- read-only EKS access
  ttl_minutes: 60
```

```bash
dfe-ops bastion up [--ttl MIN]      # read the newest AL2023 AMI, apply, wait for Online
dfe-ops bastion join [--ttl MIN]    # REFUSED until hyperi-io/culvert#40 lands
dfe-ops bastion peers               # the hub's peers and their last handshake
dfe-ops bastion hub <peer>          # reach one appliance through the tunnel
dfe-ops bastion shell               # a logged interactive shell
dfe-ops bastion forward <t> <port>  # a tunnel to a named target
dfe-ops bastion down                # revoke anything joined, then terminate
```

The four tunnel verbs exist only where a deployment runs the edge-fleet VPN, and
`hub` is the one that reaches an appliance today -- it routes the client range at
the culvert pod rather than dialling in. `join` mints an admin PEER, which
culvert drops before any rule naming it is reached, so it refuses and `down` has
nothing to revoke until it stops refusing.
[edge-vpn.md](/v/2.2.1/dfe-infra/deployment/edge-vpn.md#reaching-an-appliance-from-the-bastion) holds the
mechanism and what the appliance has to accept.

`forward` reaches a named target `bastion status` lists: the EKS API, one
`kafka` target reaching a single bootstrap broker, `kafka-iam` on MSK's own
SASL/IAM listener, and ClickHouse when the dial names a host OUTSIDE the
cluster. A Kubernetes Service name is never a target -- it resolves through
CoreDNS and the instance sits outside the cluster -- so an in-cluster
ClickHouse is reached with `kubectl port-forward` from `bastion shell`,
over the EKS API grant below. A target whose address the deployment did not
create inside the VPC -- a ClickHouse Cloud host, a Confluent Cloud or
Redpanda Cloud bootstrap -- is opened to `0.0.0.0/0` on its own port rather
than to the VPC CIDR, because a rule aimed inside the VPC leaves the forward
hanging with nothing to report it. Each target's key and port are known at plan
time, so the toolbox can be enabled in the dial before the first apply, and
reaching a broker proves reachability, TLS and SASL. A client that
must follow metadata to the other brokers -- most real Kafka clients do --
runs on the instance instead, via `bastion shell`, because the advertised
hostnames resolve only inside the VPC. AL2023 publishes no `kcat` package, so
the instance carries the Apache Kafka project's own console scripts instead
(`kafka-console-producer.sh`, `kafka-console-consumer.sh`, `kafka-topics.sh`,
`kafka-consumer-groups.sh`, `kafka-broker-api-versions.sh`), pinned to the
SAME broker version this deployment runs, plus the Java 17 runtime those
scripts need
(`terraform/modules/toolbox/aws/CONTRACT.md`, "Kafka CLI on the EC2
instance"). The container image's own `kcat` remains what the in-cluster pod
(below) uses -- Debian's repos do carry it.

Each target has its OWN Session document with the host and port fixed in the
document body -- `--parameters host=...` against one fails as an unknown
parameter -- the reason a forward cannot reach the instance's own metadata
service or an arbitrary external host. Forwarding to the EKS API also writes
a scratch kubeconfig (`.tmp/toolbox-eks-api.kubeconfig`, mode `0600`, deleted
on `down`, never merged into `~/.kube/config`).

The instance itself gets NO EKS access entry -- a forward's kubectl identity
is the operator's own role, authenticated at the laptop end of the tunnel.
`toolbox.aws.operator_role_arn` is instead granted a read-only
(`AmazonEKSViewPolicy`) access entry: read-only by default, never
cluster-admin. Starting a session at all is IAM you provide, not IAM this
module creates -- `CONTRACT.md` gives the exact policy JSON needed.

XS on the scale in [aws.md](/v/2.2.1/dfe-infra/deployment/aws.md#how-costs-are-described): one small Graviton
instance billed hourly plus its root volume, on the cluster VPC's existing NAT for its
443 egress. Zero cost while
`toolbox.enabled` is false, apart from the session-log bucket, which persists
across an up/down cycle by design.

## In-cluster pod

`helm/charts/dfe-toolbox` deploys a single-container Deployment that runs
`sleep infinity` and nothing else, on every cluster cloud or on-prem -- the
same way `network-policies` deploys, with no cloud gate. What decides whether
it does anything is `toolbox.pod.enabled`, off everywhere by default:

```yaml
toolbox:
  pod:
    enabled: false        # replicas: 0 -> 1
    kubeApiAccess: false  # a read-only Kubernetes credential, see below
    ttlSeconds: ""        # auto-scale back to zero after this many seconds
```

The dial (`deployment.yaml`'s `toolbox.pod.*`) is what turns it on -- it
reaches the chart through `argocd/appsets/layer2-platform.yaml`'s own
parameters and wins over `argocd/values/common.yaml`'s fleet-wide default.
With `enabled: true` the Deployment runs one replica; reach it with `kubectl
exec -it -n default deploy/dfe-toolbox -- /bin/sh`. It runs non-root, with a
read-only root filesystem and no capabilities; `/work` is an `emptyDir` for
anything written to disk, on top of the image's own `$HOME`/`/tmp` (above).

`kubeApiAccess: true` (refused unless `enabled` is also true) adds a
dedicated ServiceAccount and ClusterRole, bound into the DFE app namespace,
the data namespaces (`cnpg`, `strimzi`, `clickhouse`) and `otel` -- never
cluster-wide, and never `pods/exec`: exec is how an operator gets INTO a pod,
granted by the operator's own identity, never the toolbox pod's own
credential.

The chart's `NetworkPolicy` limits egress to cluster DNS and those same
namespaces, but that is a request to the CNI, not a control that enforces
itself. On EKS, the `vpc-cni` add-on now sets `enableNetworkPolicy` so the
policy is real; on-prem it depends on the cluster's own CNI supporting
`NetworkPolicy` at all -- confirm which applies before relying on the fence,
especially alongside `kubeApiAccess: true`.

`ttlSeconds`, when set, renders a companion CronJob that checks the pod's own
`status.startTime` once a minute and scales the Deployment back to zero past
that many seconds -- left empty, the pod runs until an operator disables it
by hand.

## What is recorded, and what is not

A bastion shell session (`bastion shell`) is logged to the deployment's own
session-log prefix, encrypted with the deployment's key, kept for
`toolbox.session_log_retention_days` (90 when the dial is silent) -- independent of the
telemetry dial's 2-7 day window, because a session log is evidence of human
access to a customer's private data plane, not service telemetry.

A bastion `forward` session is NOT recorded by Session Manager, and this is
not a gap in this repo's own configuration: Session Manager records no
content for a port-forward session at all, by AWS's own design. CloudTrail's
`StartSession` event, naming the per-target Session document, is the
compensating record -- because each forward target gets its own document,
the document name alone says which target was reached, whether or not
CloudTrail also logs the forward's own parameters (not verified against a
live account). The EKS control-plane audit log narrows the gap further for
Kubernetes traffic specifically: both an EKS-API forward and an in-cluster
pod's `kubectl exec` show up there, attributed to the operator's own
identity, even though the tunnel or the exec session itself carries no
transcript. Nothing does this for Kafka or ClickHouse -- a Kafka consume or
a `clickhouse-client` query through either surface leaves no record anywhere
in this deployment.