dfe-loader docs
High-performance loader from message transports (Kafka, gRPC, Memory) into ClickHouse. Point it at a topic, give it a routing rule, and it parses, routes, promotes schema columns, enriches, buffers per table, and inserts -- RowBinary by default so ClickHouse skips JSON parsing, with at-least-once Kafka delivery.
This is the index. Read architecture.md for the 10,000-foot view, clickhouse/ for the insert layer, and pipeline/ for the hot path.
What dfe-loader does for you
| You give it | It handles | You don't write |
|---|---|---|
| A Kafka topic or a gRPC push | JSON parse (sonic-rs, SIMD), zero-copy Arc<[u8]>, anything not JSON dead-lettered | A consumer, a parser, an input gate |
The routing fields (table_fields, org_routes) | Pre-flatten routing to db.table, dot-notation nested access, the default table on a miss | A router, a dispatch table |
| A ClickHouse table | Schema reflected from system.columns, fields promoted to typed columns, the rest kept in _json | A schema mapping, an ORM, a migration |
insert_format (default RowBinary) | Dynamic Map<String,Value> -> RowBinary over the HTTP Client; JSONEachRow fallback | A serialiser, a type encoder, a wire format |
capture_mode (full/raw_only/json_only/extracted_only) | _json and _raw population per table, overridable per-table and per-DDL | Conditional capture plumbing |
| Enrichment toggles (geoip/reputation/risk) | Flat enriched columns injected on the promoted row only | Lookup, caching, CIDR matching |
| Nothing extra | Per-table buffering, batch salvage, held-batch retry, schema-cache recovery, one offset commit per flush cycle | The resilience layer |
Everything else in these docs is "and here is how the pieces work".
10,000-foot view
Where things live
| Area | Doc | Covers |
|---|---|---|
| System shape | architecture.md | Layers, the dynamic insert, the client crates |
| Configuration | configuration.md | The cascade and the settings that change behaviour |
| Operations | operations.md | Running, config-check, metrics/top, hot-reload, DLQ |
| Build features | feature-flags.md | Cargo features and the ClickHouse client crates |
| Insert layer | clickhouse/ | RowBinary vs JSONEachRow, the dynamic insert, schema cache, type handling, TLS, DDL directives |
| Hot path | pipeline/ | Routing, extraction, enrichment, capture modes, buffering |
| Transports | transport/ | Kafka / gRPC / Memory, the common header |
| Deployment | deployment/ | Container + chart publishing, schemas |
| Performance | performance/ | Hot-path optimisations, parser-selection history, PGO/BOLT |
The client dependency, in one line
dfe-loader inserts through the official clickhouse crate from crates.io, with dfe-clickhouse reading the server's exception code out of a rejection. The dynamic RowBinary layer the clickhouse crate keeps out of scope -- runtime Map<String,Value> encoding -- comes from dfe-clickhouse's dynamic module. See clickhouse/clickhouse-ext.md.