Schema bootstrap -- the engine is the only controller

dfe-engine applies every ClickHouse object and every bootstrap Kafka topic a DFE deployment needs, at its own startup, from the pinned dfe-schemas wheel. No other component creates, alters or drops a database, table, view, role or topic: not a Job, not an init container, not an app at runtime, not the OTel collector.

dfe-schemas is where those objects are DEFINED. manifest.yaml lists every one of them in dependency order, and the engine iterates that list and nothing else. Adding a table is a manifest edit plus a definition file; it is never an engine release.

What runs, in what order

The phase sits inside the API lifespan, ahead of readiness (src/dfe_engine/schema/phase.py):

1. Connect, retrying an unreachable ClickHouse for DFE_ENGINE_CLICKHOUSE_BOOTSTRAP_WAIT_SECONDS (default 180). The datastore can be in the same deploy wave, so not-up-yet and broken are different answers.
2. Resolve the schema tree: the wheel inside the image, read-only, then the deployment's optional overlay directory (DFE_ENGINE_SCHEMAS_OVERLAY_DIR).
3. Render the plan for the ClickHouse platform dfe-schemas probes off the live server, unless DFE_ENGINE_CLICKHOUSE_PLATFORM pins it. Every table is an argumentless Replicated* engine with no ON CLUSTER, inside a Replicated database on self-hosted ClickHouse and the Shared database on ClickHouse Cloud. A self-hosted server without Keeper, the {cluster}, {shard} and {replica} macros or a remote_servers cluster is refused here, and so is a cluster of more than one server that keeps SQL users and roles outside Keeper. The pass fails naming what is missing, and a source deploy answers 503 clickhouse_not_replicated with the same reason. Schema DDL is sent with distributed_ddl_output_mode = throw_only_active, so a down replica catches up from the database log rather than holding the DDL until it times out. A CREATE or DROP DATABASE ON CLUSTER replays from the DDL task queue instead, so a replica down longer than its task_max_lifetime (7 days by default) misses it. Every rendered object carries a checksum taken with the platform token removed, so one schema reads as one checksum on either platform. The pattern is at https://github.com/hyperi-io/dfe-schemas/blob/main/docs/clickhouse-platform.md
4. Take the lease in schema_lock. A second replica waits for the holder to finish, then takes the lease and applies. One whose wait runs out first applies nothing, reports observed and stays NotReady, and the engine runs the pass again in the background. An expired lease is taken over. A source deploy takes the same lease around its DDL, waits for it up to DFE_ENGINE_CLICKHOUSE_BOOTSTRAP_WAIT_SECONDS, and answers 503 schema_locked when the wait runs out. Claims wait for a majority of ClickHouse replicas, so with a majority down the lease cannot be taken and the pass reports observed. A two-replica cluster cannot take the lease while either replica is down, because a majority of 2 is 2. A claim ClickHouse acknowledges 2 seconds or more after its free read is not kept and is not released either, so it holds the lease until its expiry.
5. Compare against schema_migrations and against the live catalogue, and apply what is additive.
6. Record every applied object: its kind, the dfe-schemas release, the engine release, the checksum, the statement and the action.
7. Create the topics, create-only, skipping with a logged notice where the deployment has no bus.
8. Release the lease and report.

schema_migrations and schema_lock are manifest objects like any other and the manifest declares them first, so the same pass creates them before it records anything into them. The engine carries no DDL even for its own bookkeeping.

What is applied, and what is refused

Applied automatically:

• an absent object is created
• a column the definition adds and the table lacks is added
• a JSON table's settings are brought to the platform's: MODIFY SETTING where a self-hosted table lacks write_marks_for_substreams_in_compact_parts = 0, RESET SETTING where a Cloud table carries it
• a plain view whose SELECT changed is replaced, because it holds no state
• a role, settings profile, quota or grant is re-asserted, because every statement in that set converges

Refused and named on the status route:

• a changed column type or codec, an ORDER BY, a PARTITION BY or a TTL change
• a changed materialised view, because it holds what it has already aggregated
• a live column the definition no longer declares, which is reported and never actioned

A refusal is not a failed pass: the object is named, the rest of the manifest still converges, and readiness is still reported. An operator applies the change deliberately with dfe schema apply --allow-drift, which carries out the two ClickHouse has an operation for -- a column type and a TTL. An ORDER BY or PARTITION BY change stays refused either way, because the table has to be rebuilt.

Readiness

/readyz gains a schema check beside the ClickHouse ping. Ready means the last pass converged, or the phase is switched off and gates nothing. An observed pass is not ready: the holder may still be applying. A FAILED pass leaves the pod UP and NotReady with the cause on GET /api/v1/system/schema, so an operator reads it off the API rather than off a crash loop. /livez is never consulted -- the pod is not restarted out from under whoever is reading it.

The overlay

A deployment's own schema tree is additive. It may declare objects the core does not; an object of its own whose definition sits under a core directory (tables/core/, tables/internal/, tables/otel/, tables/meta/, common-header/, registries/, roles/, topics/, views/) is refused by name and reported, and the core apply is unaffected. The overlay is optional: an install with no deploy repo boots core-only.

Dials

Variable	Default	What it does
DFE_ENGINE_CLICKHOUSE_BOOTSTRAP_TABLES	true	Apply the manifest at startup. Off reports the schema state as unknown and gates nothing.
DFE_ENGINE_CLICKHOUSE_BOOTSTRAP_WAIT_SECONDS	180	How long to keep retrying an unreachable ClickHouse before reporting the pass failed.
DFE_ENGINE_CLICKHOUSE_DDL_MAX_QUERY_SIZE	16777216	Ceiling in bytes on the max_query_size sent with schema DDL. A statement over ClickHouse's 256 KiB default -- the widest Elastic vendor table is 688 KB -- is sent with its own size plus 4 KiB, by the manifest apply and by a source deploy alike. One that would need more is refused before it is sent. A caller's own SQL never gets the setting.
DFE_ENGINE_CLICKHOUSE_PLATFORM	auto	The ClickHouse platform schema DDL is rendered for: auto probes the server, then reads the hostname. cloud or self_hosted pins it, for a server the probe reads wrongly. Anything else refuses to start.
DFE_ENGINE_KAFKA_BOOTSTRAP_TOPICS	true	Create the declared topic set at startup, create-only.
DFE_ENGINE_KAFKA_TIERED_STORAGE	false	The brokers tier to object storage, so the landing topic is created with remote.storage.enable. Off leaves the key unset rather than false, which is what a broker-level setting needs.
DFE_ENGINE_KAFKA_TOPIC_MAX_MESSAGE_BYTES	the manifest's	max.message.bytes on every topic the engine creates, this set and each source's _land and _load. Set it to the deployment's message size: a managed broker capped below the manifest's refuses the create, and the dead-letter set then holds the engine NotReady.
DFE_ENGINE_SCHEMAS_OVERLAY_DIR	unset	The deployment's additive overlay tree.

The two bootstrap dials are separate rather than one: they fail for different reasons, and a deployment with an external broker wants one off and the other on.

A stack that writes to ClickHouse with the engine switched off is unsupported. No engine, no tables.

Operator surface

• GET /api/v1/system/schema -- the last pass, object by object: state, versions, platform, per-object action and checksum, the refused list, the overlay refusals, the topics created
• GET /api/v1/system/version -- carries the dfe-schemas release as schemas
• dfe schema plan -- what an apply would do, changing nothing
• dfe schema apply [--allow-drift] -- the boot phase by hand, under the same lease
• dfe schema status -- the ledger: which release each object on this cluster came from
• dfe_schema_bootstrap_state (0 unknown, 1 converged, 2 failed, 3 running, 4 observed), dfe_schema_bootstrap_duration_seconds, dfe_schema_version_info, dfe_schema_objects_refused, dfe_schema_lease_release_failures_total (a pass that applied but could not release the lease, which then lapses at its expiry)

What the engine still renders itself

Per-SOURCE objects are defined at use time from an operator's own config, so they are not manifest data:

• a source's table and its standard views, rendered by src/dfe_engine/schema/schema_builder_v2.py with the engine form and platform settings dfe-schemas supplies
• an operator's own quota tiers and the per-org and per-group ClickHouse users, reconciled by src/dfe_engine/governance/ch/

Both go through SchemaApplier, so the one DDL shape cannot be bypassed. tests/unit/test_schema/test_one_ddl_gate.py fails the build on any rendered table that is not Replicated* and any table statement carrying ON CLUSTER. A guard test (tests/unit/test_schema/test_schema_control_rules.py) fails the build on a CREATE TABLE, CREATE DATABASE or CREATE MATERIALIZED VIEW anywhere under src/ outside dfe_engine/schema/, and on a literal that qualifies an object the manifest does not declare.