# Architecture

> What culvert is made of, why it is one container instead of a stack, and the
> invariants that no single file makes obvious. Look a fact up here before you
> change the startup order, the firewall chains or the PKI paths.

## The short version

A VPN server is easy to install and hard to keep correct. The usual answers
either mutate a host with a shell script, or ship a management plane, a
database and a web UI to configure one daemon. Culvert takes the other route:
one image, configuration by environment variable, and every capability past
plain OpenVPN over UDP an explicit opt-in. Upgrading is a `docker pull`.

That choice is what shapes everything below. There is no state store, so a
replica is defined entirely by its environment and its mounted volumes. There
is no control plane, so the container itself has to refuse to start when a
control it was asked for did not install.

## What is in the image

The base is `ubuntu:26.04`, pinned by digest. Every third-party binary comes
from upstream directly, and each arrives with its own verification:

| Component | Source | How it is verified |
|---|---|---|
| OpenVPN 2.7+ | The project's signed apt repo | Downloaded keyring must hold exactly one primary key, whose fingerprint is pinned. Installed version then checked against a floor |
| easy-rsa, iptables, wireguard-tools, stunnel4 | Ubuntu archive | apt, over the base image's trust |
| openvpn-auth-oauth2 | Pinned GitHub release `.deb` | SHA256 per architecture, checked before `dpkg` |
| wstunnel | Pinned GitHub release tarball | SHA256 per architecture, checked before extract |
| Python dependencies | `requirements-docker.txt`, exported from `uv.lock` | `pip --require-hashes`, so a drifted release fails the build |

Counting `pub` records rather than `fpr` records on the OpenVPN keyring is
deliberate. A key with signing subkeys emits one `fpr` line per subkey, all
legitimately the same primary, so counting fingerprints would reject a valid
key and counting keys catches a second one smuggled in alongside the real one.

The apt repo serves one version per suite and cannot be pinned, which is why
the version floor is asserted after install rather than requested before it.
The server config depends on 2.7 behaviour, so building on 2.6 would change it
silently. `openvpn-auth-oauth2` gets the artefact treatment instead because its
apt repo serves `releases/latest` and cannot be pinned at all.

DCO is supported but its DKMS package is deliberately absent. Inside a
container DKMS would compile against kernel headers that are not there, and the
module has to be loaded on the host regardless -- `vm-setup/setup-host.sh` does
that. Leaving it out drops a compiler toolchain from the image at no cost to
the capability.

## One entrypoint, one ordered sequence

`scripts/entrypoint.py` is a thin orchestrator. Every piece of real work lives
in a focused module under `scripts/lib/`, and the entrypoint's job is the
order.

```text
config -> directories -> log rotation -> network -> routing control
       -> forward guards -> [OpenVPN branch] -> [WireGuard branch]
       -> observability -> CRL refresh -> client download -> block
```

The OpenVPN branch initialises PKI, renders the UDP, HTTPS and TCP server
configs from `config/*.template`, sets up stunnel, generates any auto-clients
and starts the OIDC helper. The WireGuard branch brings up `wg0` through
`wg-quick`, starts wstunnel and starts a connection monitor. `CULVERT_PROTOCOL`
selects one, the other, or both, and `both` runs both branches in full.

In WireGuard-only mode there is no OpenVPN process to block on, so the
entrypoint marks itself started and ready and waits on signals. The kernel
interface is already up at that point, which is why readiness flips there
rather than in the server starter.

## PKI: two modes, and local disk is the source of truth

Local mode has Easy-RSA generate a CA and a server certificate. External mode
fetches them through scalo secrets from a file mount, OpenBao or AWS Secrets
Manager. In both modes the files under the PKI directory are what OpenVPN
reads, and a remote provider is the update mechanism rather than the authority.
That is what lets an external deployment keep serving when the provider is
unreachable, on cached certificates, with a warning.

Three things here are not guessable from the code's shape:

- `easyrsa init-pki` is never called. All it creates is three directories,
  and it insists on removing the PKI directory first, unconditionally on 3.2.x
  with no opt-out. The directory is a volume mount, so the removal fails
  outright, and where it could succeed it would take the CA and every
  certificate issued from it. 3.1.x's `init-pki soft` is not a portable
  substitute either, because 3.2.x reads that argument as a curve name.
  Culvert creates the directories itself, and `build-ca` still refuses to
  overwrite an existing CA, so a half-built PKI fails loudly instead of being
  replaced.
- The `tls-crypt-v2` server key is minted locally only when no provider
  supplies one. A client's key derives from it, so two servers with
  different keys reject each other's clients. Once
  `CULVERT_SECRETS_TC_KEY_PATH` is set, a failed fetch is fatal rather than a
  fall back to minting, because an install that comes up healthy and rejects
  clients at random is worse than one that refuses to start.
- CRL refresh differs by mode and cannot be shared. Local PKI regenerates
  from its own CA. External PKI has no CA key present, so the only way to pick
  up a revocation is to re-fetch. OpenVPN re-reads the CRL on every handshake,
  so a new file needs no reload. The server reloads only when the new CRL
  revokes a certificate the old one did not, to end that session before it
  renegotiates, and that reload drops every connected client.

A CRL that reaches its `nextUpdate` makes OpenVPN refuse every client,
including valid ones. The refresher therefore reports remaining life on every
cycle, not just whether the last refresh worked.

## Enforcement is two iptables chains, and the order matters

NAT and MSS clamping go in the base `FORWARD` and `POSTROUTING` paths.
Everything policy-shaped lives in two dedicated chains, both rebuilt on each
start so restarts cannot stack duplicates:

- `CULVERT_FWD` -- opt-in routing control. Client isolation, the egress
  allow-list, and the downstream-admin gate that lets an operator initiate back
  into the tunnels.
- `CULVERT_GUARD` -- always on by default. Drops forwarded client traffic to
  link-local, which on a cloud instance is the metadata service and this host's
  credentials.

Four ordering rules hold this together, and each exists because the obvious
arrangement is wrong:

1. `CULVERT_GUARD` is installed AFTER `CULVERT_FWD`. Both insert at `FORWARD`
   position 1, so the last one installed is evaluated first, and the guard has
   to be. It is separate from routing control precisely because routing control
   is opt-in and the guard must hold on a default configuration.
2. `CULVERT_FWD` is detached from `FORWARD` before it is flushed, built
   completely, and jumped at last. A flushed chain that is still jumped falls
   through to the `FORWARD` policy, so there must be no window where it is
   jumped and empty.
3. Client-to-client rules sit above the conntrack accept, so cross-tunnel
   traffic is decided without reference to state and a RELATED packet cannot
   slip between clients.
4. Reverse-admin replies get a NAT `RETURN`, scoped to established flows only.
   Without it the general masquerade rewrites the client's reply to the
   server's own address and the exchange never completes. Scoping it keeps a
   client that initiates towards the admin range masqueraded.

Rules of this kind are not advisory, so a failed install raises and the
container exits. Serving clients while reporting a control that is not there is
the outcome being avoided. Chain creation and rule deletion stay tolerant,
because both are expected to fail on a normal restart.

The guard is scoped to `FORWARD` deliberately, which leaves the server's own
access to the metadata service intact -- external PKI on AWS authenticates with
exactly those instance credentials. The tunnels carry IPv4 only and only
`net.ipv4.ip_forward` is enabled, so there is no ip6tables setup at all.

## Observability is one listener

`CULVERT_METRICS_ADDR` (default `0.0.0.0:9090`) carries `/livez` and `/readyz`
always, and `/metrics` when metrics are enabled. OTLP is a push to a collector,
so it adds no listener. Health and metrics deliberately share the port so that
one published port covers the operator surface and nothing about it rides on
the VPN listeners.

## Deployment artefacts are generated, not authored

`deploy/deployment-contract.json` and the compose fragment are rendered from
culvert's scalo deployment contract by `scripts/generate-deploy-artefacts.py`.
Editing either by hand puts it out of step with the contract, and a unit test
compares the committed output against a fresh render. The release assembles the
Helm chart from the contract file on the scalo-service library chart, which
renders every contract field: the WireGuard port gate, the PKI claim, the root
identity, the capabilities and the single replica. The chart's `version` and
`appVersion` come from the release, so the chart and the image it deploys move
together. No chart is committed to this repository.

## Invariants, collected

- Configuration is `CULVERT_*` environment variables through scalo's cascade,
  with profile YAML as an extra source. Explicit env always beats a profile.
- `/etc/vpn` and `/var/log/vpn` are the canonical paths and nothing aliases
  them. Compatibility symlinks to the legacy OpenVPN directories were removed
  rather than forced, because the package already owns those as real
  directories and the link landed inside one, which silently defeated PKI
  persistence.
- A replica holds no state beyond its volumes, which is what makes horizontal
  scaling possible and what makes a shared `tls-crypt-v2` key mandatory for it.
- Defaults are generic, never site-specific. Anything site-shaped belongs in a
  profile or the deployment values.