# dfe-fetcher architecture

> The repo is a Cargo workspace: one app crate and a source framework split into
> an I/O-free core plus one leaf crate per shape family, with the file tailer
> vendored from Vector under `third-party/`. Every crate is `publish = false` and
> inherits its version from `[workspace.package]`, so a release bump touches the
> root manifest only. Diagrams (system, the tick sequence, the profile grammar,
> the crate DAG) live in [internals.md](/v/2.2.2/dfe-fetcher/internals.md); this file is the
> codemap, the invariants and the build graph.

## Codemap

| Where | What lives there |
|---|---|
| `crates/core` (`dfe-fetcher-core`) | The row model (`Row`, `Mark`, `RowSource`, `UnitSpec`, `TickCtx`), the batcher, the compiled per-row rules (CEL filter, routes, added fields), the snapshot envelope and its reassembler, the checkpoint contract (`CursorStore`, `CursorValue`, mark folding), the once-resolved secret cell (the resolver is the I/O crate's), the framework's error type and the list of metric names. No HTTP, no transport, no filesystem. |
| `crates/rest` (`dfe-fetcher-rest`) | The declarative REST profile grammar and its validation, `{{ cel }}` templates, the runtime axes (auth modes, pagers, streaming decoders, the request executor with retry and its per-unit origin gate), the hooks (row builders and the S3 lister), the queue shape, and `RestShape`, the `RowSource` for a bound profile. |
| `crates/db` (`dfe-fetcher-db`) | The `Store` contract (dump, keyset tail), the `sources.db` grammar, `DbShape` (the `RowSource` over an instance's stores), the per-dialect keyset predicate, the blocking-cursor pump and the block-to-row framing that leases every buffered block. Engines are features: `odbc` (unixODBC + `arrow-odbc` typing + `arrow-json` writing), `clickhouse` (server-side `JSONEachRow`, `dfe-clickhouse` for the ping and typed exceptions) and `mongodb` (the official driver: `find` to dump, change stream or `_id` to tail). |
| `crates/file` (`dfe-fetcher-file`) | The `FileSource` contract, the `sources.file` grammar, the dump reader (NDJSON, JSON array, CSV, gzip by magic) and the tail spec; the tailer sits behind the `tail` feature over the vendored crates. |
| `third-party/vector-file-source{,-common}` | Vector's file tailer (MIT), vendored and depended on by path from `crates/file` only; provenance in `third-party/README.md`. |
| `crates/fetcher` (`dfe-fetcher`, the binary) | The config cascade and hot reload, the typed source blocks, their registry and their mapping onto profile instances (`config/builtin.rs`), the scheduler, `Driver` (the tick the scheduler runs for every framework shape), the emitter, the pipeline (enrichment, DLQ, readiness), the output transports, the cursor file store, the shipped REST profiles (`profiles/*.yaml`), the deployment contract and capability catalog, the ingest server and the container and Vector extractors. |
| `config.example.yaml`, `docs/` | Operator-facing surface, at the repo root; the app crate's tests reach them through the manifest directory. |

Adding a REST source is a profile (YAML) plus an instance in `sources.rest`;
adding a hook is a variant on an axis enum in `crates/rest`; adding an engine is
a feature-gated module in `crates/db`. A typed source block (`sources.github`,
`sources.aws`) keeps its struct for the operator and maps onto an instance of
the shipped profile of the same name at load; the driver runs it like any
`sources.rest` entry.

## Negative invariants

- `dfe-fetcher-core` depends on no other workspace crate and on scalo only for
  `expression`. It never names an HTTP client, a transport or a file. Async
  appears only as `futures-core` stream and future types; tokio is linked for
  its clock types and its once-cell and never spawns a task there.
- `dfe-fetcher-rest`, `dfe-fetcher-db` and `dfe-fetcher-file` depend on `core`
  and never on each other or on the app. None sends to a transport, writes a
  cursor or reads the config cascade; each receives bound units and yields rows.
- The app never re-implements a shape, a pager, a decoder or an auth mode; it
  owns wiring, transports, checkpoint persistence, config and the deployment
  contract. There is no per-provider Rust in the app: a provider is a profile,
  and what a profile cannot say is a hook on an axis in `crates/rest`.
- No crate depends on a Vector crate from a registry. The file tailer is the
  vendored copy under `third-party/`, depended on path-only by
  `dfe-fetcher-file`.
- Nothing deployment-specific: no hostnames, addresses, vault paths or licence
  detail outside `config.example.yaml` placeholders and tests.

## Build graph

`core` is the root and compiles in seconds (serde, bytes, chrono, uuid,
smallvec, csv-core, cel via scalo `expression`). `rest` (reqwest,
async-compression, backon, reqsign), `db` and `file` are siblings off `core`
and compile in parallel. The cold-compile critical path is unchanged from the single-crate
layout: scalo with `transport-kafka` (librdkafka) and `transport-grpc` (tonic,
prost), then the app's link. A profile or pager change rebuilds `rest` and the
app; an engine bump rebuilds `db` alone. `db` is always linked for its grammar
(a few serde types), so a `sources.db` block naming an engine the binary lacks
is refused at load with the feature to build; the engines themselves and the
file tailer are off the default build.

Feature flags on the app follow scalo's `<capability>-<backend>` naming:
`db-odbc`, `db-clickhouse`, `db-mongodb`, `file`, `file-tail`, plus `jemalloc`
and `full`. `.hyperi-ci.yaml` lists feature sets explicitly so a runner is
never asked for a native driver it does not carry; `db-odbc` needs unixODBC on
the host. The release build adds `db-odbc`, and its image carries unixODBC and
the open-licence drivers (docs/reference/db-drivers.md).