# Data plane

> The close-coupled source - ingest - schema - transform path. One source
> definition carries routing, schema binding, and field-map bindings together;
> the receiver, loader, views, hunts, and query API all read the same
> definition, so there is no separate ETL config to drift. The system map is
> [../architecture.md](/v/2.2.2/dfe-engine/architecture.md).

```mermaid
flowchart LR
    SRC[source YAML] --> RCV[receiver] --> LDR[loader] --> CH[(ClickHouse)]
    CH --> VIEWS[standard views] --> HUNTS[hunt runner] & QAPI[query API]
```

## Documents in this area

| Doc | Covers |
|---|---|
| [source.md](/v/2.2.2/dfe-engine/data-plane/source.md) | Source as the top-level data abstraction |
| [source-definition.md](/v/2.2.2/dfe-engine/data-plane/source-definition.md) | the source YAML field by field, and what a write is validated against |
| [source-registry.md](/v/2.2.2/dfe-engine/data-plane/source-registry.md) | states, storage backends, CRUD, topics on deploy, the API surface |
| [source-flow.md](/v/2.2.2/dfe-engine/data-plane/source-flow.md) | INPUT, optional TRANSFORM, OUTPUT on the Kafka or gRPC transport; what compiles from a source; how a change reaches a pod |
| [source-routing.md](/v/2.2.2/dfe-engine/data-plane/source-routing.md) | the exact receiver and loader config a source compiles into |
| [topic-contract.md](/v/2.2.2/dfe-engine/data-plane/topic-contract.md) | the Kafka topics a source implies: ensure, converge, status, remove, and the dials |
| [external-pipelines.md](/v/2.2.2/dfe-engine/data-plane/external-pipelines.md) | the customer's own Vector, Fluentd or Logstash as a source's origin or transform on Kafka |
| [source-detections.md](/v/2.2.2/dfe-engine/data-plane/source-detections.md) | rules, hunts and Sigma views on a source |
| [schema.md](/v/2.2.2/dfe-engine/data-plane/schema.md) | schema type system: primitives, attributes, use cases, DDL |
| [schema-classes.md](https://github.com/hyperi-io/dfe-engine/blob/v1.22.90/docs/data-plane/schema-classes.md) | engine class reference for the schema system (SchemaLoader, SchemaBuilderV2, DDL generation) |
| [schema-sync.md](https://github.com/hyperi-io/dfe-engine/blob/v1.22.90/docs/data-plane/schema-sync.md) | engine<->loader runtime schema contract |
| [field-mapping.md](/v/2.2.2/dfe-engine/data-plane/field-mapping.md) | standards translation layer (Sigma / ECS / CIM views) |
| [sigma.md](/v/2.2.2/dfe-engine/data-plane/sigma.md) | Sigma rule-provider pipeline (feeds -> catalogue -> propagation -> source-views) + query cost leaderboard |
| [expressions-cel.md](/v/2.2.2/dfe-engine/data-plane/expressions-cel.md) | CEL profile, classifier tiers, SQL transpile |
| [query-api.md](/v/2.2.2/dfe-engine/data-plane/query-api.md) | query API surface, and the [in-image Python client](/v/2.2.2/dfe-engine/data-plane/query-api-python.md) |
| [hunt-runner-scaling.md](/v/2.2.2/dfe-engine/data-plane/hunt-runner-scaling.md) | pull-based runner, deterministic-due scaling, keda-shim fail-safe |
| [hunt-match-delivery.md](/v/2.2.2/dfe-engine/data-plane/hunt-match-delivery.md) | where a hunt match is written, who sees it, alert destinations and what the runner sends |
| [hunt-schedule-smoothing.md](/v/2.2.2/dfe-engine/data-plane/hunt-schedule-smoothing.md) | phase-drift load smoothing (incl. the reference derivation) |