# Storage: the deploy-time matrix

> REFERENCE. Look up one cell: a service, a deployment mode and a storage model,
> and what that combination does. Where DFE's data physically lives is not one
> choice but several independent ones, and this page is the complete list of them
> -- which combinations ship, which are refused, and how strong the evidence is
> behind each. Every future addition claims a cell here rather than adding a
> paragraph somewhere.

The deep dives stay where they are: [clickhouse.md](/v/2.2.1/dfe-infra/deployment/clickhouse.md) for the
ClickHouse target matrix, [clickhouse-tiering.md](/v/2.2.1/dfe-infra/deployment/clickhouse-tiering.md) for the
hot/cold walkthrough, [kafka/README.md](/v/2.2.1/dfe-infra/deployment/kafka.md) for the broker
defaults.

## The four dimensions

1. **Service** -- `clickhouse` or `kafka`. Both take a storage model; they do not
   take the same ones.
2. **Deployment mode** -- who deploys the datastore or broker.
   ClickHouse: `cluster` (operator), `single` (plain StatefulSet), `external`
   (bring your own). Kafka: `disabled` (no broker, gRPC transport), `single`
   (non-operator broker), `cluster` (Strimzi or Redpanda operator), `external`.
3. **Storage model** -- where the data lives, and whether it moves or is copied.
4. **Sub-dimensions** -- the dials inside a model: cache placement, object-store
   flavour, request bounds, per-topic granularity.

Modes and models are INDEPENDENT axes. Not every pair is legal, and the illegal
ones fail the render with a named guard rather than deploying something inert.

## The vocabulary

A storage model is named `<family>-<bulk>`, and both halves carry meaning.

| Half | Value | Means |
| --- | --- | --- |
| family | `tiered` | Data MOVES to the bulk store. The bulk copy is the only copy, so both stores must be durable. |
| family | `cached` | Data is COPIED to the bulk store. The local copy is disposable; the bulk store holds the durable one. |
| bulk | `block` | A second volume on a cheaper StorageClass. |
| bulk | `object` | An S3-compatible object store. |

`local` is the fifth value and the default: one volume, no bulk store, nothing
extra rendered.

The charts read the two halves through a pair of helpers
(`storageFamily` / `storageBulk` in each chart's `_storage.tpl`), so every
template conditional is a single comparison and a new model claims a cell
without touching them.

## Status and evidence vocabularies

Every cell below carries one status and one proof level. Both lists are closed.

| Status | Means |
| --- | --- |
| **default** | What a deployment gets at that tier with no override. |
| **opt-in** | Shipped and supported, inert unless the deployer sets it. |
| **refused** | The render fails. The guard is named in the cell. |
| **follow-up** | Understood and written up, deliberately not surfaced as a value. |
| **wired-unverified** | A code path exists but has never run against the real backend. |

| Proof | Means |
| --- | --- |
| **live-proven** | Run on real hardware, with the measurement named. |
| **render-verified** | `helm template` plus an assertion in `scripts/tests/test_storage_model.py`. No runtime. |
| **unverified** | Neither. |

## Dimension 2 x 3: ClickHouse

`clickhouse.storageModel` in `helm/charts/clickhouse-cluster`.

| Mode | `local` | `cached-object` | `tiered-block` |
| --- | --- | --- | --- |
| `cluster` | **default** (scale tier), live-proven | **opt-in**, render-verified; the disk-ordering constraint is live-proven | **opt-in**, **live-proven** |
| `single` | **default** (single tier), live-proven | **opt-in**, render-verified | **opt-in**, render-verified |
| `external` | **opt-in**; the model is forced, the supplied ClickHouse owns its storage | **refused** | **refused** |

**Preference order.** ClickHouse Cloud runs SharedMergeTree by default and is
the reference shape. `cached-object` is our best-effort equivalent of it, and
is the preferred model wherever an object store exists: always on Kubernetes in
a cloud, and on-prem whenever an S3-compatible store such as MinIO is supplied.
`local` is the last resort -- on-prem with no object store, and the slim and
single tiers, where one volume is the point. A deployment migrating off the
retired private-cloud ClickHouse fork lands here too -- see
[Migrating from DFE 2.x before 2.2](/v/2.2.1/dfe-infra/deployment/upgrades.md#migrating-from-dfe-2x-before-22).

That order is a three-value dial: `clickhouse.storageModel` takes `auto` (the
chart default, and what an empty value means too), `cached-object` or `local`.
On `auto`, `dfe-clickhouse.storageModel` reads `cached-object` from a set
`clickhouse.objectStore.endpoint` and `local` from nothing; the other two are
explicit overrides that win outright -- `cached-object` refuses to render with
no endpoint set, `local` keeps local storage even with one configured. Every
template reads the derived answer, so the two cannot disagree; because every
model is locked at first deploy, the derivation runs once and
`governance/policies/storage-layout.yaml` holds it afterwards. The `cluster`
row above still reads `local` as default because `cached-object` is
render-verified only; it becomes the cluster-tier default once live-proven.

The dial reaches the chart from `sizing.storage_model` in the deployment dial
(`deployment.example.yaml`, default `auto`) -- the `dfe-ops init` wizard asks
for it directly, and `resolve_sizing.py` writes `clickhouse.storageModel` into
the scale-tier values fragment only on an explicit value, leaving the key out
on `auto` so the chart derives. Scale-tier only (`resolve_sizing.py` sizes
that tier alone); slim and single set no `clickhouse.storageModel` in
`argocd/values/profile-{slim,single}.yaml`, so they already derive the same
way with no dial needed.

**Where the AWS bucket comes from, and what it commits you to.**
`terraform/modules/kubernetes-cluster/aws/object-store.tf` provisions the S3
bucket `cached-object` needs on AWS -- SSE-KMS on the deployment's own key,
ACLs disabled, all public access blocked, plaintext HTTP denied, no versioning,
a lifecycle rule aborting an incomplete multipart upload after 7 days -- and
mints a Pod Identity role scoped to that bucket alone, bound to the chart's own
`dfe-clickhouse` ServiceAccount (`clickhouse.serviceAccount.name`) rather than
the release namespace's default account, so nothing else running there
inherits the role. `bootstrap.sh` carries
the bucket's URL onto the cluster secret as the
`dfe.hyperi.io/clickhouse_object_store_endpoint` annotation, and
`argocd/appsets/layer2-data.yaml` passes it through as
`clickhouse.objectStore.endpoint`. Set
`clickhouse.objectStore.usePodIdentity: true` alongside it and the pod
authenticates as that role instead of the static-key ExternalSecret every
other target uses (Dimension 4: credential binding, below).

The cache placement the resolver decides travels the same way, because a fresh deploy repo has no overlay for it to arrive in: `resolve_sizing.py` writes `sizing/<tier>.clickhouse-storage.json`, `bootstrap.sh` carries it as `dfe.hyperi.io/clickhouse_storage`, and layer2-data hands clickhouse-cluster its `storageModel` and `objectStore` keys. They land with the bucket or not at all, the model always beside an instance-store cache, so the cache never arrives without an object store to front. With a bucket on the cluster secret they beat a deploy-repo overlay, as values fixed at first deploy should. With none, nothing storage-related is inlined, and an overlay's own `clickhouse.objectStore.endpoint` stands, which is how an on-prem MinIO reaches the chart.

That bucket is unconditional, so **every AWS deploy derives `cached-object`**
-- which is the preference order above applied, not an accident, but it is not
the opt-in the ClickHouse table still records. The model is locked at first
deploy, so the first AWS apply is what live-proves it; until then AWS is the
one target where the table's `opt-in` reads as `derived`. A deployment that
wants `local` on AWS has to say so, before the first deploy: `sizing.storage_model:
local` in the deployment dial (the wizard asks for it), or
`clickhouse.storageModel: local` in the deploy repo's own
`infra/clickhouse-cluster.yaml` overlay for a deployment resolve_sizing.py
never sizes.

The two refusals share one guard in `dfe-clickhouse.validateStorageModel`:

```text
clickhouse.storageModel=cached-object is meaningless with mode=external --
the supplied ClickHouse owns its own storage
```

**`cached-object` evidence.** The cache disk must sort AFTER the disk it wraps,
because ClickHouse builds a cache only once its backing disk exists and the
operator serialises `extraConfig` as sorted-key JSON. Naming it anything earlier
fails the server at startup with `BAD_ARGUMENTS` "there is no such disk (it
should be initialized before cache disk)" -- proven live 2026-08-30. The disk
names `s3_object` and `s3_object_cache` encode that ordering.

**`tiered-block` evidence.** Demotion triggers on the hot volume's free space as
a fraction of its total. Verified on a cluster whose hot tier was an SSD with
55.7 GiB free of 61 (a free ratio of 0.91) and whose cold tier was a separate
491 GiB spinning disk: `moveFactor` 0.95 demoted new parts to the cold volume,
0.2 left them hot. One node, one dataset, changing nothing but the value.

**Single mode carries the same on-disk layout.** In cluster mode the operator
mounts each additional volume claim at `/var/lib/clickhouse/disks/<name>` and
registers a ClickHouse disk of the same name; single mode runs no operator, so
the chart mounts the same path and declares the disk itself. Both modes then
share one `storage_configuration` fragment.

## Dimension 2 x 3: Kafka

`kafka.storageModel` in `helm/charts/kafka`. `tiered-object` is KIP-405 tiered
storage, which is what the Kafka and Strimzi documentation calls it.

| Mode | `local` | `tiered-object` |
| --- | --- | --- |
| `disabled` | **default** (slim tier) | **refused** -- "needs kafka.mode=cluster" |
| `single` | **default** (single tier) | **refused** -- same guard |
| `cluster` + `strimzi` | **default** (scale tier) | **opt-in**, render-verified |
| `cluster` + `redpanda` | **opt-in** (BSL gate) | **refused** -- "is a Strimzi Kafka CR field" |
| `external` | **opt-in** | **refused** -- needs `mode=cluster` |

Two further guards on the opt-in cell. `kafka.tieredObject.className` is
required, because Strimzi supports `type: custom` only and ships no
RemoteStorageManager -- the broker image must carry a plugin. And
`kafka.operatorVersion` must be at least 0.38.0, where
`spec.kafka.tieredStorage` landed: Strimzi drops unknown CR fields silently, so
an older operator would deploy a broker that looks configured and tiers nothing.

## Dimension 4: cache placement (`cached-object`, ClickHouse)

`clickhouse.objectStore.cache.volume` in `helm/charts/clickhouse-cluster`.

| Placement | Status | Proof | Constraint |
| --- | --- | --- | --- |
| `pvc` -- a subdirectory of the data PVC, bounded by `clickhouse.objectStore.cache.maxSizeRatioToTotalSpace` (0.6), or by `clickhouse.objectStore.cacheSize` when that is set | **default** | render-verified | The cache and the part metadata share `clickhouse.storage.size`, and ClickHouse refuses both bounds at once |
| `instance-store` -- an `emptyDir` on the node's local NVMe, limited to `clickhouse.objectStore.cacheSize` (required), with the cache's `max_size` at `cache.maxSizeRatioToSizeLimit` (0.95) of it | **default** on AWS, where the resolver places it on every NVMe ClickHouse shape, and **opt-in** elsewhere, cluster mode only | render-verified | Needs `clickhouse.mode: cluster` and a node whose NVMe is assembled under the kubelet |
| Dedicated disposable cache PVC | **follow-up** | unverified | A different mechanism to `instance-store` above -- see below |

**Why `instance-store` over a bigger `pvc`.** A gp3 volume provisioned large
enough to hold a useful cache is the expensive answer, and it is never what
either dial value defaults to. On AWS the Karpenter `clickhouse` pool's shape
(`shapes/compute-shapes.yaml`) already carries local NVMe sized for exactly
this cache: at about 474 GB an r9gd.2xlarge's NVMe costs what gp3 costs
provisioned to the instance's own EBS ceiling, with 14.5x the IOPS.
`instance-store` mounts that NVMe as a plain pod-level `emptyDir` at
`/var/lib/clickhouse-cache` (see "Extra mounts" below), with a
`nodeSelector` on `dfe.hyperi.io/workload: clickhouse` -- the label
`build_pool_classes` (`scripts/resolve_sizing.py`) gives every pool it
builds, named after the pool's own use case. The EKS managed node group for
the same shape carries that label too, and assembles its NVMe the same way:
the resolver marks the shape `instance_store_policy: RAID0`, and the node
group's launch template hands nodeadm a NodeConfig asking for the RAID0 array
under the kubelet and containerd, as Karpenter's `instanceStorePolicy` does.
Because the array also holds images and pod logs, and the kubelet evicts at
10% free on it, the resolver sets `cacheSize` to
`clickhouse.instance_store_cache_share` (0.8) of the NVMe rather than all of
it. The data PVC keeps its default mount and now holds only the `s3_object` metadata_path. The resolver's own
`clickhouse.storage.size` default does not shrink to reflect that yet -- it is
still sized for a PVC-resident cache -- so a deployer running `instance-store`
can safely set it much smaller by hand; teaching the resolver that ratio is a
follow-up.

**The failure mode is the point, not a gap.** An `emptyDir` is destroyed with
its pod: a node replacement -- Karpenter consolidation, an instance
replacement, a rolling restart -- empties the cache, and the server rewarms it
from the object store cold on the next read. The `clickhouse` Karpenter pool is
on-demand only (`STATEFUL_USE_CASES` in `scripts/resolve_sizing.py`), so spot
reclamation is not a cause here, but node churn still is -- which is exactly
why that pool already carries a disruption budget of one node at a time:
losing more than one node's cache at once is the outage the budget buys
against, not something `instance-store` is meant to prevent outright.

**Extra mounts stay outside the data volume.** The container runtime creates a nested mount's missing directories root-owned inside the data PVC, and the non-root server cannot create anything else in them. A cache mounted at `/var/lib/clickhouse/disks/s3_object_cache` left `disks/` root-owned, so every replica exited unable to create the `s3_object` `metadata_path`. Hence `instance-store` mounts at `/var/lib/clickhouse-cache`, and `test_the_cache_mount_sits_outside_the_data_volume` fails a render that nests one.

A dedicated cache PVC hits the same wall: the operator mounts each additional claim at `/var/lib/clickhouse/disks/<name>` and registers a disk of that name, which also rules out `s3_object_cache`.

## Dimension 4: object-store flavour

| Flavour | Service | Status | Proof |
| --- | --- | --- | --- |
| S3 | ClickHouse | **opt-in** | render-verified |
| GCS via its S3-compatible endpoint | ClickHouse | **opt-in** | unverified. GCS rejects the batch-delete call, so set `clickhouse.objectStore.supportBatchDelete: false` |
| Azure Blob | ClickHouse | **follow-up** (no value surfaced) | wired-unverified |
| `local_blob_storage` | ClickHouse | **follow-up** (no value surfaced) | live-proven on a rig |
| Whatever the plugin speaks | Kafka | **opt-in** | unverified |

Azure Blob is addressed with `storage_account_url`, `container_name` and
`account_name` / `account_key`. Authentication and addressing were confirmed
against the Azurite emulator; container creation could not be, because Azurite
3.36.0 rejects the create-container call from ClickHouse and from `az` alike.
Provisioning the container ahead of time is the better posture regardless, since
the credential then needs no container-create right.

`local_blob_storage` presents a plain filesystem path through the object-storage
interface, which makes it cacheable -- the route to `cached-block` with no
object store at all. Verified on a rig: the wrapped disk resolved as
`ObjectStorage / Local`, and a warm scan served 3.34 MB from cache against 79 KB
from source.

Kafka's flavour is not a chart dimension. `kafka.tieredObject.config` passes
plugin keys through verbatim, so the plugin owns the addressing that the
ClickHouse chart owns itself.

## Dimension 4: request bounds (`cached-object`, ClickHouse)

`clickhouse.objectStore.retryAttempts`, `.connectTimeoutMs` and
`.requestTimeoutMs` are **opt-in** and unset by default, leaving the server's own
behaviour alone. That behaviour is the reason they exist: **live-proven**, a read
against an unreachable store blocked for over nine minutes before failing, and
`1 / 2000 / 5000` brought the same failure down to 10.4 seconds. Those numbers
are a demonstration, not a recommendation -- one retry is aggressive against real
S3, which returns transient 5xx routinely, so pick production values against the
actual store. What they must not stay is unbounded, because a nine-minute stall
on a dashboard is indistinguishable from an outage.

## Dimension 4: credential binding (both services)

| Cell | Status | Proof |
| --- | --- | --- |
| Derived path `<project>/<env>/clickhouse/s3` and `<project>/<env>/kafka/tiered` | **default** | render-verified |
| Rebind with `objectStore.remoteKey`, `.accessKeyProperty`, `.secretKeyProperty`, `.secretStoreName` | **opt-in** | render-verified |
| A credential literal in a values file | **refused** by construction | render-verified |

Both charts read the credential from the environment, wired from a Secret the
secrets store materialises. Nothing is inline in the CR or the ConfigMap, both of
which live in git.

## Dimension 4: granularity of the opt-in

| Cell | Status | Proof |
| --- | --- | --- |
| Kafka per-topic: the landing topic gets `remote.storage.enable: true`, DLQ topics do not | **default** under `tiered-object` | render-verified |
| ClickHouse per-table | **follow-up** | The policy is server-wide, so no table DDL is needed |

Without the per-topic switch the brokers hold the plugin and move nothing. On the
ClickHouse side, age-based demotion (`TTL ts + INTERVAL 7 DAY TO VOLUME 'slow'`)
is per-table DDL and belongs to dfe-engine's governance, not to this chart.

## Cells the vocabulary names and no chart builds

Both are refused at render time, so asking for one fails loudly rather than
deploying a datastore that ignores the request.

| Cell | Service | What it would be | Blocking work |
| --- | --- | --- | --- |
| `tiered-object` | ClickHouse | Parts MOVED to an object store, matching Kafka's shape | A ranked policy whose cold volume is an object disk; no evidence gathered |
| `cached-block` | ClickHouse | An SSD cache over a cheap local volume | Needs `local_blob_storage`, which is proven on a rig and not surfaced |

## Locked at deploy

Every model is a one-way decision, enforced twice.

The **mechanisms**: the ClickHouse operator takes no new disk on an existing
`ClickHouseCluster`, so a model that adds a volume claim cannot be reached later.
And `volumeClaimTemplates` are immutable, so a size or class change needs every
PVC patched and the StatefulSet recreated. `local-path` cannot resize at all.

The **enforcement**: dfe-engine seeds a `storage-layout` policy holding
`clickhouse.mode`, `clickhouse.storageModel`, `clickhouse.objectStore.*`,
`clickhouse.tieredBlock.*`, `clickhouse.storage.size`,
`clickhouse.storage.storageClass` and the five matching `kafka.*` patterns as
protected vars. The API refuses a post-deploy edit and names the pattern that
blocked it. A holder of `helmvars:override` can still make a deliberate
exception, and a direct git commit still gets through -- the policy governs the
API, not the repo.

The API also reports what syncing a write would do, and the verdict differs by
service: writing `kafka.storageModel` is a `roll`, because Strimzi reconciles
`tieredStorage` onto the running CR, while `clickhouse.storageModel` is a
`recreate`.

## Runtime identity: what a rename is not

Some names in the rendered output are not labels -- they are what the running
server resolves its data through. Renaming one is a rebuild and a restore, not a
values edit, so they stay fixed while the values vocabulary above them changes.

| Identifier | Why it is frozen |
| --- | --- |
| Disks `s3_object`, `s3_object_cache`; policy `s3_cached` | ClickHouse refuses a policy change that drops a volume name it already loaded, and `merge_tree.storage_policy` is the server-wide default every existing table resolved through |
| Policy `default`, volume `default`, the cold volume name | The same refusal, and the cold name is also the PVC template name, which is immutable in both the operator claim template and a StatefulSet |
| Mount path `/var/lib/clickhouse/disks/<name>` | Changing it strands the metadata the disk points at |
| Derived credential paths `<project>/<env>/clickhouse/s3` and `<project>/<env>/kafka/tiered` | A deployment that seeded there and never set `remoteKey` would lose its credential |
| Secret names `dfe-clickhouse-s3`, `dfe-kafka-tiered` | Moving them orphans the materialised Secret and rolls every pod that reads it |

## The docker path is a different axis

`dfe-docker` has no storage models and no size dial. It has ONE storage choice
and it is a LOCATION: set `DFE_DATA_ROOT` and every named volume becomes a bind
onto `${DFE_DATA_ROOT}/<name>`.

State it as asymmetry rather than parity: **Kubernetes locks size and class;
docker locks which disk.** Two consequences carry over from
`dfe-docker/docs/operating.md`: switching an existing deployment does not migrate
data between locations, and `make clean` removes the volume objects while the
bind directories keep their contents, so reclaiming the space is an explicit
delete of `${DFE_DATA_ROOT}`.

## Where the pieces live

- ClickHouse chart: `helm/charts/clickhouse-cluster`, models in
  `templates/_storage.tpl`.
- Kafka chart: `helm/charts/kafka`, models in `templates/_storage.tpl`.
- Assertions: `scripts/tests/test_storage_model.py` -- run it with
  `python3 scripts/tests/test_storage_model.py`.
- The protected-var lock: dfe-engine
  `src/dfe_engine/governance/resources/policies/storage-layout.yaml`.