Skip to main content

Deploying, and moving to a newer stack

A repeatable deploy turns ONE file and pins ONE version. This covers both, plus what an upgrade actually does to a running stack.

Running the thing day to day is operating.md.

The deployment dial: one file the deploy reads​

A repeatable deploy turns ONE file. deployment.example.yaml is the template; copy it to deployment.yaml (gitignored, never committed) and populate it. That copy is the SSoT for the deploy -- registry, pinned version, service footprint, host exposure and the broker -- and make dial renders its docker-vm slice into .env. A redeploy is a dial edit, not a hunt through .env:

make dial && make stack && make ci

make dial writes the deploy-controlled keys over the .env that make init generated, make stack pins the certified image set from the dial's version.pin, and make ci pulls and starts it. Both read the registry anonymously, so neither needs a login.

Two properties keep the file safe to hand around. Secrets are references, never values -- secrets.backend and secrets.ref name WHERE the deployment's secrets live (OpenBao by default), and whoever deploys resolves them, so the dial itself carries no secret. Estate endpoints stay blank -- an empty endpoints.clickhouse_host means "use the in-stack container", and you set it (or let the caller inject it) only to point the engine at an external warehouse.

This file is the docker-vm SLICE. The canonical superset -- docker-vm plus the Kubernetes and cloud dials, and the schema -- lives in dfe-infra, but a lone dfe-docker clone deploys from its own deployment.yaml alone, with no dfe-infra checkout needed.

Staying current: pinned or track-latest​

A box stays current one of two mutually exclusive ways. make modes states both and reports which one THIS checkout is on -- ask the stack rather than this page, for the same reason make limits exists.

make modes                                          # the contract + this box's mode
python3 ops/daemon-update/self_update.py --dry-run  # the live latest-vs-applied

Pinned (default). make stack VERSION=X.Y.Z && make ci pins the whole certified set from the signed stack-manifest and starts it. It hard-fails rather than ever pull latest. make dial sets the pin from the deployment dial's version.pin, so a redeploy is a dial edit plus make dial && make stack && make ci. This is the production-safe default: nothing moves until you move it.

Per image, the pin is an explicit release tag plus the digest that tag resolved to (e.g. DFE_ENGINE_VERSION=v1.17.13@sha256:50c4428...), so a retagged image cannot change what a pinned box runs. That now covers the hyperdx fork too. Where the SSoT has no digest for an image the render emits a bare tag instead, which downgrades the pin without announcing it. The registries also publish latest and sha-<commit> tags for every image, the hyperdx fork included -- none of them is ever consumed here, so what a release publishes and what a deploy runs stay two separate decisions.

Track-latest (opt-in). ops/daemon-update/install.sh installs a systemd timer that discovers the newest certified stack tag and runs the SAME make stack + make ci, but only when something newer has shipped. It never pulls latest. By default it tracks STABLE releases only; DFE_UPDATE_ALLOW_PRERELEASE=1 (commented in the service unit) also takes -rc builds. See ops/daemon-update/README.md.

Latest (development only). make stack VERSION=latest pins the newest published stack, read anonymously from the OCI registry, and moves forward on every re-run; DFE_STACK_REPIN_IMAGES=1 also repins each DFE image at its GitHub Latest release. It is not a third deploy mode: make modes reports it as LATEST rather than PINNED. Do not deploy off it -- pin a version.

The daemon fast-forwards the checkout before pinning, because a stack version is images plus the compose that runs them. It refuses rather than pull over uncommitted changes to tracked files, and DFE_UPDATE_SKIP_GIT_PULL=1 turns the refresh off for a checkout managed another way. On the pinned path that is your job: git pull before make stack, or you get new images under an old compose file.

The two are mutually exclusive: a track-latest box lets the daemon own the version, so leave version.pin out of the dial there. make modes calls it AMBIGUOUS if it finds both set.

Until a GA stack ships, the only published tag is a pre-release, so a stable-only track-latest daemon finds nothing to apply -- pin explicitly, or set DFE_UPDATE_ALLOW_PRERELEASE=1 knowingly.

Updating a daemon VM without sudo​

A box on the track-latest daemon has two users. The service user (dfe) runs the timer and owns the checkout, and the operator logs in as someone else. Left alone, every git and make step then needs sudo -u dfe, which an unattended agent cannot use (git runs the repo's config as the target user).

install.sh sets the box up so the operator never needs it:

  • Start an update on demand. A polkit rule lets members of the service user's group start, stop and restart dfe-docker-update.service and dfe-docker-proof.service. systemctl start dfe-docker-update.service runs the timer's tick as dfe and waits for it. Add --no-block to return at once and follow with journalctl -u dfe-docker-update -f. The tick takes the newest tag in the signed stack-manifest repository, not a git branch, so a cycle moves the version by publishing one and starting the unit.
  • Work in the checkout directly. The checkout is group-writable with the setgid bit, UMask=0002 makes the service's own files match, and .env is group read/write and closed to others. The operator can git pull and start the proof with systemctl start dfe-docker-proof.service as themselves, and read it with journalctl -u dfe-docker-proof from the adm group. Without console TLS they can also run make stack VERSION=X.Y.Z, make ci and make post directly. With it, make stops, because the proxies run as the user who runs make and DFE_PROXY_CERT_DIR is readable by the service user only, so those targets run through the units or as the service user. Use this path for a version the manifest does not list as newest.

Secret files go no wider than group read/write. .env, .env.*, access-summary.md and everything in env/ are never open to others, and a rewrite by either user keeps the group-write bit it finds.

Converting an existing install. Once, as a user with sudo, from the box's own checkout:

sudo -u dfe git -C /opt/dfe-docker pull --ff-only
sudo DFE_DOCKER_DIR=/opt/dfe-docker DFE_UPDATE_USER=dfe DFE_OPERATOR_USER=ubuntu \
  DFE_UPDATE_ENABLE_TIMER=0 /opt/dfe-docker/ops/daemon-update/install.sh

DFE_UPDATE_ENABLE_TIMER=0 leaves the timer as it is, so an operator-triggered box stays that way. The operator logs in again to pick up the group. Check it from that login, with no sudo:

id -nG                                   # includes dfe
git -C /opt/dfe-docker pull --ff-only
systemctl start --no-block dfe-docker-update.service && systemctl status dfe-docker-update.service

Starting the unit is the check for the polkit rule. A refusal prints Interactive authentication required. pkcheck --detail answers only to root, so it cannot test the rule from the operator's login.

python3 ops/daemon-update/share_checkout.py --dry-run --repo-dir /opt/dfe-docker lists what the conversion would change and touches nothing.

Upgrading an existing deployment​

Two changes bite an existing stack. Neither is silent if you read this; both are silent if you do not.

ClickHouse moved to a named volume. Its data previously lived in the container's writable layer and died with docker compose down. It now mounts clickhouse-data:/var/lib/clickhouse. On the first make ci after upgrading, the container is recreated against an empty volume -- anything still in the old writable layer is not migrated and will appear to have vanished. Export it before upgrading if it matters.

Most published ports moved from 0.0.0.0 to 127.0.0.1. Anything reaching this box from elsewhere -- a remote clickhouse-client, a Prometheus scrape of :9090-9096, a colleague's browser -- gets connection refused with no hint as to why. DFE_BIND_HOST=0.0.0.0 restores the old behaviour, having read the auth section above.

The DFE UI moved from 0.0.0.0 to 127.0.0.1 as well. Every web UI now binds DFE_BIND_SCOPE, which defaults to localhost for a developer workstation, so :3000 stops answering from off the box. A VM or small deploy sets DFE_BIND_SCOPE=all (or docker.bind_scope: all in the deployment dial) to publish the UIs on every interface -- operating.md.

CLICKHOUSE_PASSWORD is now generated, not blank. make init mints one, so an upgraded stack that runs make init (to pick up new .env.example keys) gets a real password. ClickHouse stores the default-user credential in its data volume on first init, so a pre-existing clickhouse-data volume still expects the OLD (blank) password and the loader/engine/HyperDX now authenticate with the new one -- every query fails Authentication failed. Two ways through:

  • Keep the old behaviour: set CLICKHOUSE_PASSWORD= (blank) in .env before starting. make init only tops up a MISSING key, so an explicit blank is kept.
  • Adopt the password: set the ClickHouse default user to the generated value once (ALTER USER default IDENTIFIED BY '<value>' against the running instance, or recreate the volume if it holds nothing you need), then restart the stack.

A fresh deployment has neither problem -- the volume is created with the generated password from the start.