# Kubernetes

> Not a profile - a deploy shape, and a walkthrough you follow top to bottom.
> Each release publishes a Helm chart beside the image, at
> `oci://ghcr.io/hyperi-io/charts/culvert`. The chart version is the release
> version, so chart `X.Y.Z` deploys image `vX.Y.Z`. Nothing is cloned to
> install it.

The chart runs one replica. The contract marks culvert a singleton, so it
renders a Recreate rollout, no autoscaler and no disruption budget. WireGuard
mints its server key inside the pod, and a second pod would issue client
configs the first rejects.

## 1. A namespace the pod is allowed in

The VPN data plane creates the tun device, programs routing and NAT, and sets
`net.ipv4.ip_forward`. That needs root, `NET_ADMIN` and the `/dev/net/tun` host
device, so a restricted-PodSecurity namespace rejects it. Give culvert its own
namespace at the privileged level:

```bash
kubectl create namespace vpn
kubectl label namespace vpn pod-security.kubernetes.io/enforce=privileged
```

## 2. Install

The chart renders the pod from culvert's contract and nothing more. The tun
device, `ip_forward`, the config file and the opt-in listeners all come from
values. Write this as `values.yaml`:

```yaml
# Mount `config` as the single file /etc/vpn/profiles/culvert.yaml, so the
# image's own profiles stay beside it. CULVERT_PROFILE makes culvert read it.
configMount:
  subPath: true

extraEnv:
  CULVERT_PROFILE: culvert
  CULVERT_SERVER_CN: vpn.example.com
  CULVERT_LOG_MODE: stdout

# /proc/sys is read-only inside a container, so ip_forward is set by a short
# privileged init container. It changes the pod's network namespace only.
initContainers:
  - name: enable-ip-forward
    image: '{{ include "scalo-service.image" . }}'
    command: [sysctl, -w, net.ipv4.ip_forward=1]
    securityContext:
      privileged: true
    resources:
      requests:
        cpu: 10m
        memory: 16Mi

# OpenVPN and WireGuard open /dev/net/tun.
extraVolumes:
  - name: tun
    hostPath:
      path: /dev/net/tun
      type: CharDevice
extraVolumeMounts:
  - name: tun
    mountPath: /dev/net/tun

# LoadBalancer Services for every port marked `public`.
publicService:
  enabled: true
  externalTrafficPolicy: Local

# Opt-in listeners. Keep the entries whose CULVERT_* switch you turn on.
extraPorts:
  - { name: openvpn-tcp, port: 1194, protocol: TCP, public: true }
  - { name: openvpn-https, port: 443, protocol: TCP, public: true }
  - { name: wg-https, port: 4443, protocol: TCP, public: true }
  - { name: oauth2-udp, port: 9000, protocol: TCP, public: true }
  - { name: oauth2-https, port: 9001, protocol: TCP, public: true }
  - { name: oauth2-tcp, port: 9002, protocol: TCP, public: true }
  - { name: client-download, port: 8443, protocol: TCP, public: true }
```

Then install the release you want. `helm show chart
oci://ghcr.io/hyperi-io/charts/culvert` prints the newest:

```bash
helm install culvert oci://ghcr.io/hyperi-io/charts/culvert \
  --version <release> -n vpn -f values.yaml
```

Four notes on the values:

- `config:` is written to `culvert.yaml`. Put anything you would put in a
  profile there. Without `CULVERT_PROFILE: culvert` the file is never read.
- `extraEnv` takes every `CULVERT_*` knob from [.env.example](https://github.com/hyperi-io/culvert/blob/v2.1.30/.env.example)
  that is not a secret. Secrets go in a Secret (see section 5).
- `image: '{{ include "scalo-service.image" . }}'` gives the init container the
  pod's own image, so it never drifts from the release.
- Helm replaces a list rather than merging it. A second values file that sets
  `extraVolumes` or `extraVolumeMounts` has to repeat the tun entries.

On a kubelet started with `--allowed-unsafe-sysctls net.ipv4.ip_forward` you can
drop the init container and set `podSecurityContext.sysctls` instead.

## 3. The PKI volume

The chart renders a `culvert-pki` PersistentVolumeClaim (1Gi, ReadWriteOnce)
by default and mounts it at `/etc/vpn/pki`. It holds the CA, the CRL and the
WireGuard peer allocations. Lose it and every issued client config stops
working, because the next start mints a new CA.

The claim names no StorageClass, so it takes the cluster default. On a cluster
with no default StorageClass the claim stays `Pending` and the pod waits on it.
Name one, or bring your own claim:

```yaml
writablePaths:
  pki:
    persistence:
      storageClass: fast-ssd   # or: existingClaim: my-pki
      size: 1Gi
```

`persistence.enabled: false` swaps the claim for an `emptyDir`. Every restart
then mints a new CA. Use it for a throwaway test and nothing else.

## 4. Expose and open listeners

The chart renders a ClusterIP Service for the pod: the metrics port `9090` and
every VPN port. `publicService.enabled` adds LoadBalancer Services for the
ports marked `public: true`:

- `culvert-public` carries the TCP ports and `culvert-public-udp` the UDP ones,
  because many clouds refuse one Service that mixes the two.
- The metrics port is not public, so the LoadBalancer never publishes `/livez`,
  `/readyz` or `/metrics`.
- `externalTrafficPolicy: Local` keeps the client's real source address, which
  per-client routing rules and the connection log need.

Pin the addresses clients dial with `publicService.loadBalancerIP` (TCP) and
`publicService.udpLoadBalancerIP` (UDP), and clamp who can reach them with
`publicService.loadBalancerSourceRanges`. Or leave `publicService` off and put
a Gateway API `UDPRoute` (or TLS passthrough for the HTTPS listeners) in front
of the ClusterIP Service.

OpenVPN over UDP (`1194/UDP`) is always on. Every other listener is two edits
that must agree: the `CULVERT_*` switch in `extraEnv`, and its `extraPorts`
entry so the port reaches the Service.

| Port name | Port | Switch | Notes |
|-----------|------|--------|-------|
| `openvpn-tcp` | 1194/TCP | `CULVERT_TCP_ENABLED=true` | TCP fallback |
| `openvpn-https` | 443/TCP | `CULVERT_HTTPS_ENABLED=true` | OpenVPN over HTTPS (stunnel) |
| `wg-https` | 4443/TCP | `CULVERT_WG_HTTPS_TUNNEL_ENABLED=true` | WireGuard over HTTPS (wstunnel) |
| `oauth2-udp` | 9000/TCP | `CULVERT_OAUTH2_ENABLED=true` | OIDC callback for the UDP listener |
| `oauth2-https` | 9001/TCP | `CULVERT_OAUTH2_ENABLED=true` | OIDC callback for the HTTPS listener |
| `oauth2-tcp` | 9002/TCP | `CULVERT_OAUTH2_ENABLED=true` | OIDC callback for the TCP listener |
| `client-download` | 8443/TCP | `CULVERT_CLIENT_DOWNLOAD_ENABLED=true` | Needs `CULVERT_CLIENT_DOWNLOAD_BIND=0.0.0.0` and a bearer token |

WireGuard is the exception. Set `config.protocol` to `wireguard` or `both` and
the chart opens `51820/UDP` itself, with no `extraPorts` entry.

## 5. OIDC and other secrets

The OIDC client secrets come from a Secret the chart reads by name,
`culvert-oauth2` unless `secrets.oauth2.existingSecret` says otherwise:

```bash
kubectl -n vpn create secret generic culvert-oauth2 \
  --from-literal=CULVERT_OAUTH2_CLIENT_SECRET=... \
  --from-literal=CULVERT_OAUTH2_HTTP_SECRET=...
```

Both keys are optional, so a server with OIDC off starts without the Secret.
The non-secret OIDC settings (`CULVERT_OAUTH2_ISSUER`, `CULVERT_OAUTH2_CLIENT_ID`
and the rest in [.env.example](https://github.com/hyperi-io/culvert/blob/v2.1.30/.env.example)) go in `extraEnv`. The callbacks
also need a TLS pair at `CULVERT_OAUTH2_TLS_CERT` and `CULVERT_OAUTH2_TLS_KEY`,
which are required while OIDC is on. Mount a Secret there with `extraVolumes`
and `extraVolumeMounts`.

Any other secret, such as the client-download token, comes in from a Secret too:

```yaml
extraEnv:
  CULVERT_CLIENT_DOWNLOAD_TOKEN:
    valueFrom:
      secretKeyRef:
        name: culvert-download
        key: token
```

`extraEnvFrom` loads a whole Secret as environment when you would rather.

## 6. External PKI

Production clusters generally do not want a self-generated CA. Set
`CULVERT_PKI_MODE=external` and point at one of:

- File: certs mounted from a Kubernetes Secret (or cert-manager) volume.
  `CULVERT_SECRETS_*_PATH` are file paths. The values below do this.
- OpenBao: Kubernetes auth exchanges the pod ServiceAccount token for a
  short-lived token, with no static secret in env.
- AWS Secrets Manager: use IRSA on EKS, with no static AWS keys.

For the file provider, create the Secret, then add this to `values.yaml`:

```bash
kubectl -n vpn create secret generic culvert-pki-external \
  --from-file=ca.crt --from-file=server.crt --from-file=server.key \
  --from-file=crl.pem --from-file=tc.key
```

```yaml
extraEnv:
  CULVERT_PKI_MODE: external
  CULVERT_SECRETS_PROVIDER: file
  CULVERT_SECRETS_CA_CERT_PATH: /etc/vpn/pki-external/ca.crt
  CULVERT_SECRETS_SERVER_CERT_PATH: /etc/vpn/pki-external/server.crt
  CULVERT_SECRETS_SERVER_KEY_PATH: /etc/vpn/pki-external/server.key
  CULVERT_SECRETS_CRL_PATH: /etc/vpn/pki-external/crl.pem
  CULVERT_SECRETS_TC_KEY_PATH: /etc/vpn/pki-external/tc.key

# These replace the lists from section 2, so they carry the tun entries too.
extraVolumes:
  - name: tun
    hostPath:
      path: /dev/net/tun
      type: CharDevice
  - name: pki-external
    secret:
      secretName: culvert-pki-external
      defaultMode: 0400
extraVolumeMounts:
  - name: tun
    mountPath: /dev/net/tun
  - name: pki-external
    mountPath: /etc/vpn/pki-external
    readOnly: true
```

`ca.crt`, `server.crt` and `server.key` are required. `crl.pem` and `tc.key` are
optional, so drop their paths if the Secret does not carry them. With no
`tc.key` the server mints its own tls-crypt-v2 key, and a client config only
works against the server that issued it.

The fetched files are checked as one set before any is written: the server
certificate must verify against `ca.crt` and be in date, `server.key` must be its
key, `crl.pem` must be signed by `ca.crt`, and `tc.key` must be a tls-crypt-v2
server key. If any check fails, nothing is written and the server starts on the
files already in its PKI volume.

Detail is in [External PKI in the README](https://github.com/hyperi-io/culvert/blob/v2.1.30/README.md#external-pki) and the
External PKI section of [.env.example](https://github.com/hyperi-io/culvert/blob/v2.1.30/.env.example).

## 7. Observe

- **Probes:** liveness `/livez`, readiness `/readyz` and a startup probe on
  `/livez`, all on the metrics port (`9090`). They are served always,
  independent of the metrics toggle.
- **Metrics:** the pod carries `prometheus.io/scrape`, `prometheus.io/port` and
  `prometheus.io/path` annotations. `/metrics` serves when
  `CULVERT_METRICS_ENABLED=true`.
- **Drain:** on SIGTERM readiness drops at once, and connected clients keep
  their tunnels for `CULVERT_DRAIN_SECONDS` (10 by default) while load balancers
  stop sending new ones. The drain ends early once no client is connected.
  OpenVPN is then stopped, which tells UDP clients to reconnect, and every child
  gets 15 seconds to exit, all inside a 30 second termination grace period.

## See also

- [use-case-edge-fleet.md](/v/2.2.2/culvert/use-case-edge-fleet.md) - the hub-and-spoke fleet
  shape that runs on this chart
- [use-case-corporate.md](/v/2.2.2/culvert/use-case-corporate.md) - OIDC SSO and per-user
  certs, the config side of a corporate cluster deploy
- [Deployment in the README](https://github.com/hyperi-io/culvert/blob/v2.1.30/README.md#deployment) and
  [Observability](https://github.com/hyperi-io/culvert/blob/v2.1.30/README.md#observability)
- [addressing.md](/v/2.2.2/culvert/addressing.md) - tunnel subnet defaults and overrides