# DFE documentation 2.2.2 > The Data Fusion Engine (DFE) is a product suite an organisation deploys to run its own data pipeline: events in over HTTP, through Kafka or gRPC, into ClickHouse. Release 2.2.2. Every prose page has a markdown copy at its URL with .md on the end, and https://dfe.docs.hyperi.io/latest/ always serves the newest release. https://dfe.docs.hyperi.io/latest/openapi/dfe-engine.json is the engine's HTTP API spec. ## Table of Contents - [Data Fusion Engine overview](https://dfe.docs.hyperi.io/latest/overview.md): Land it. Use it. Scale it. The Data Fusion Engine (DFE) does what you used Splunk, Elastic and a lakehouse for, without the limitations. Every sour... - [Using these docs with an AI agent](https://dfe.docs.hyperi.io/latest/ai-agents.md): Point your agent at https://dfe.docs.hyperi.io/llms.txt. - [Coming from a lakehouse](https://dfe.docs.hyperi.io/latest/coming-from-lakehouse.md): If you've run Snowflake, Databricks, BigQuery, or Athena over Iceberg or Delta tables, you'll bring a warehouse's assumptions about how data lands ... - [Coming from Splunk or Elastic](https://dfe.docs.hyperi.io/latest/coming-from.md): If you've run Splunk, Elastic or OpenSearch, or several of them, you'll bring assumptions about how a data platform works. DFE can take the place o... - [Culvert documentation](https://dfe.docs.hyperi.io/latest/culvert.md): Start with the README for what culvert is and the five-minute `docker run`. This directory is everything past that. - [Addressing](https://dfe.docs.hyperi.io/latest/culvert/addressing.md): Which IP range the tunnels use, why the default is what it is, and when to change it. If you followed a link here from one of the use-case docs, th... - [Architecture](https://dfe.docs.hyperi.io/latest/culvert/architecture.md): What culvert is made of, why it is one container instead of a stack, and the invariants that no single file makes obvious. Look a fact up here befo... - [Performance, tuning, and the cipher stance](https://dfe.docs.hyperi.io/latest/culvert/performance.md): culvert's defaults err high-security first and resilient-everywhere second. This note explains what actually moves the needle for a client on a hos... - [Corporate access](https://dfe.docs.hyperi.io/latest/culvert/use-case-corporate.md): Per-user certificates plus human SSO against your identity provider, group gating, and a TCP fallback for hostile guest networks. This is the... - [Edge fleet -> receiver](https://dfe.docs.hyperi.io/latest/culvert/use-case-edge-fleet.md): An appliance fleet streaming data into a central receiver, with reverse admin back down the same tunnels. Any hub-and-spoke telemetry fleet has thi... - [Home / lab access](https://dfe.docs.hyperi.io/latest/culvert/use-case-home.md): Reach your home LAN, or a throwaway VM, from the outside. OpenVPN over UDP, split tunnel, a self-generated local CA - nothing else. This is the... - [Kubernetes](https://dfe.docs.hyperi.io/latest/culvert/use-case-kubernetes.md): Not a profile - a deploy shape, and a walkthrough you follow top to bottom. Each release publishes a Helm chart beside the image, at... - [Travel / hostile networks](https://dfe.docs.hyperi.io/latest/culvert/use-case-travel.md): For networks that block or throttle VPNs. Culvert can run listeners that look like ordinary HTTPS, so protocol blocking does not see a VPN on the w... - [Culvert Client Setup Guide](https://dfe.docs.hyperi.io/latest/culvert/vpn-client-setup.md): This guide covers installing and configuring OpenVPN and WireGuard clients on Linux, macOS, and Windows. - [dfe-archiver documentation](https://dfe.docs.hyperi.io/latest/dfe-archiver.md): How dfe-archiver is built, and how to run it. - [dfe-archiver architecture](https://dfe.docs.hyperi.io/latest/dfe-archiver/architecture.md): Where things live and which way the dependencies point. The design itself -- the pipeline, the rolling model, the storage backends -- is internals.md. - [Archiving](https://dfe.docs.hyperi.io/latest/dfe-archiver/archiving.md): The archiver keeps a compressed copy of every record it receives, in files you can read with standard tools. This page is for the person running it... - [DFE Archiver Design Document](https://dfe.docs.hyperi.io/latest/dfe-archiver/internals.md): High-volume Kafka-to-storage archiver designed for PB/s scale data pipelines. - [dfe-deploy documentation](https://dfe.docs.hyperi.io/latest/dfe-deploy.md): The architecture of dfe-deploy. - [dfe-deploy architecture](https://dfe.docs.hyperi.io/latest/dfe-deploy/architecture.md): Why this repo is shaped the way it is, and the rules that are enforced rather than written down and hoped for. The per-directory READMEs carry the ... - [dfe-docker documentation](https://dfe.docs.hyperi.io/latest/dfe-docker.md): Running DFE on a single host with Docker Compose: a first look, deploying and upgrading, day-to-day operation, development, and what to do when som... - [How the Compose stack fits together](https://dfe.docs.hyperi.io/latest/dfe-docker/architecture.md): This repo is **packaging, not product**. It ships a `docker-compose.yml`, the config files the components mount, and Python helpers that resolve a ... - [Common failures](https://dfe.docs.hyperi.io/latest/dfe-docker/common-failures.md): Each section is one symptom and what causes it. If none matches, read the stack's state first: troubleshooting. The open problems whose symptoms mi... - [Configuration reference](https://dfe.docs.hyperi.io/latest/dfe-docker/configuration.md): Lookup, not narrative. Every variable the stack reads, what it does, and what it defaults to. .env.example is the annotated template these come from. - [Deploying, and moving to a newer stack](https://dfe.docs.hyperi.io/latest/dfe-docker/deploying.md): A repeatable deploy turns ONE file and pins ONE version. This covers both, plus what an upgrade actually does to a running stack. - [Taking DFE for a first look](https://dfe.docs.hyperi.io/latest/dfe-docker/evaluating.md): **This applies to you if** you want to see DFE actually work - on your laptop, a demo box, or a proof of concept - without committing to anything yet. - [Known issues](https://dfe.docs.hyperi.io/latest/dfe-docker/known-issues.md): These are real and open. Each one has a symptom that misleads. - [How the stack reports on itself](https://dfe.docs.hyperi.io/latest/dfe-docker/observability.md): Three surfaces, three different questions. - [Operating the DFE Compose stack](https://dfe.docs.hyperi.io/latest/dfe-docker/operating.md): This applies to you if people other than you depend on this stack -- a single box, an edge site, or a partner deployment. Small is still production. - [Profiles](https://dfe.docs.hyperi.io/latest/dfe-docker/profiles.md): This repo owns every profile in `service_profiles.yaml`. Nothing generates them. - [Troubleshooting the DFE Compose stack](https://dfe.docs.hyperi.io/latest/dfe-docker/troubleshooting.md): This applies to you if a stack is misbehaving and you need to find out why -- yours or someone else's. - [dfe-engine documentation](https://dfe.docs.hyperi.io/latest/dfe-engine.md): The DFE engine's documentation: the architecture and data evolution, plus one directory each for the control plane, data plane, deployment and refe... - [DFE architecture](https://dfe.docs.hyperi.io/latest/dfe-engine/architecture.md): DFE is a deployable product suite: event-data ingest, ClickHouse storage, scheduled detection, and exploration, all governed through one git-backed... - [The DFE Control-Plane Layer](https://dfe.docs.hyperi.io/latest/dfe-engine/control-plane.md): The **control-plane layer** is the small set of repos that own the DFE control plane - configuration, RBAC, schema, the operator UI, and the deploy... - [Identity provider setup](https://dfe.docs.hyperi.io/latest/dfe-engine/control-plane/idp-setup.md): One page per identity provider, written for the customer's IdP admin. The DFE operator sends the admin the page for their IdP plus two values: the ... - [Microsoft Entra ID sign-in for DFE](https://dfe.docs.hyperi.io/latest/dfe-engine/control-plane/idp-setup/entra.md): DFE asks for two things from your Microsoft Entra ID tenant: your users sign in to DFE with their work account, and DFE reads which security groups... - [Google Workspace sign-in for DFE](https://dfe.docs.hyperi.io/latest/dfe-engine/control-plane/idp-setup/google.md): DFE asks for two things from your Google Workspace: your users sign in to DFE with their Google account, and DFE reads which Google groups each use... - [Okta sign-in for DFE](https://dfe.docs.hyperi.io/latest/dfe-engine/control-plane/idp-setup/okta.md): DFE asks for two things from your Okta org: your users sign in to DFE with their Okta account, and DFE reads which Okta groups each user is in, so ... - [OIDC + RBAC architecture and design](https://dfe.docs.hyperi.io/latest/dfe-engine/control-plane/oidc-rbac-architecture.md): The single source of truth for how dfe-engine handles authentication, external OIDC, group->role mapping, the two RBAC planes, the management API a... - [RBAC - the engine's own reads](https://dfe.docs.hyperi.io/latest/dfe-engine/control-plane/rbac-engine-reads.md): Section 5.3 of rbac.md, which keeps its number so section references resolve. - [RBAC - IdP group sync and links](https://dfe.docs.hyperi.io/latest/dfe-engine/control-plane/rbac-groups.md): Section 4.4 of rbac.md, which keeps its number so section references resolve. - [RBAC - identity resolution and first login](https://dfe.docs.hyperi.io/latest/dfe-engine/control-plane/rbac-sign-in.md): Sections 2.1 and 3.5 of rbac.md, which keep their numbers so section references resolve. - [RBAC vocabulary - roles, groups and terms](https://dfe.docs.hyperi.io/latest/dfe-engine/control-plane/rbac-vocabulary.md): One vocabulary, used verbatim by every moving part: dfe-engine, dfe-ui, dfe-infra, dfe-deploy, dfe-schemas, and the test environments. If a role or... - [RBAC, Multi-Tenant ClickHouse, and Auth Architecture](https://dfe.docs.hyperi.io/latest/dfe-engine/control-plane/rbac.md): **Scope:** Auth, granular RBAC, multi-tenant CH with row-level security, OIDC group sync, dfe-hyperdx integration, Argo CD RBAC export - [Synthetic Data](https://dfe.docs.hyperi.io/latest/dfe-engine/control-plane/synthetic-data.md): Realistic synthetic events on demand, for two jobs: a continuous live-looking inbound stream for sales demos, and repeatable data for testing. Neve... - [Data evolution](https://dfe.docs.hyperi.io/latest/dfe-engine/data-evolution.md): DFE is a data evolution platform by design. Data lands by default and it just works. You improve and evolve from there. - [Data plane](https://dfe.docs.hyperi.io/latest/dfe-engine/data-plane.md): The close-coupled source - ingest - schema - transform path. One source definition carries routing, schema binding, and field-map bindings together... - [DFE Expression Language — CEL](https://dfe.docs.hyperi.io/latest/dfe-engine/data-plane/expressions-cel.md): DFE uses **CEL (Common Expression Language)** for conditions across the Python engine, the Rust services and the CLI. CEL is not Turing-complete: n... - [External pipelines](https://dfe.docs.hyperi.io/latest/dfe-engine/data-plane/external-pipelines.md): You already run Vector, Fluentd, Fluent Bit or Logstash, and you don't want to rebuild it. You don't have to. Your pipeline plugs into a DFE source... - [Field mapping layer](https://dfe.docs.hyperi.io/latest/dfe-engine/data-plane/field-mapping.md): A generic, standard-agnostic layer translating security/observability standards (Sigma, Elastic ECS, Splunk CIM) to DFE schema columns. Queries wri... - [How a hunt match reaches a person](https://dfe.docs.hyperi.io/latest/dfe-engine/data-plane/hunt-match-delivery.md): A match is a row in a ClickHouse table. When the hunt names an alert destination, the runner also sends one message for each fire that matched. - [Hunt-runner scaling: deterministic-due, no engine dependency](https://dfe.docs.hyperi.io/latest/dfe-engine/data-plane/hunt-runner-scaling.md): The hunt runner is pull-based and multi-pod. **Default deployment is ONE steady pod** (no ScaledObject at all) - the runner is a thin orchestrator ... - [Hunt schedule smoothing - flattening CH load by phase drift](https://dfe.docs.hyperi.io/latest/dfe-engine/data-plane/hunt-schedule-smoothing.md): How DFE keeps a fleet of recurring detection queries from stacking up into CPU and memory peaks on a shared ClickHouse cluster, by learning the loa... - [DFE Query API - Python client](https://dfe.docs.hyperi.io/latest/dfe-engine/data-plane/query-api-python.md): `dfe_engine.query.QueryClient` ships inside the dfe-engine image. dfe-engine is not published to PyPI, so code outside the image calls the HTTP API... - [DFE Query API](https://dfe.docs.hyperi.io/latest/dfe-engine/data-plane/query-api.md): A label-based query interface over ClickHouse parameterised views, with multi-tenant isolation, scope-based RBAC and native JSON responses. - [Schema bootstrap -- the engine is the only controller](https://dfe.docs.hyperi.io/latest/dfe-engine/data-plane/schema-bootstrap.md): dfe-engine applies every ClickHouse object and every bootstrap Kafka topic a DFE deployment needs, at its own startup, from the pinned... - [Schema Type System v2](https://dfe.docs.hyperi.io/latest/dfe-engine/data-plane/schema.md): The schema YAML SSoT is the dfe-schemas repo (the version-pinned `dfe-schemas` wheel), registries included, and the engine keeps no copies. - [Sigma detection pipeline](https://dfe.docs.hyperi.io/latest/dfe-engine/data-plane/sigma.md): Sigma is off by default: every path under `/api/v1/sigma` answers 404 until the engine runs with `sigma.enabled` set, which is... - [The source definition](https://dfe.docs.hyperi.io/latest/dfe-engine/data-plane/source-definition.md): One YAML file per source, and it carries the whole stream: identity, where the records come from, whether they are transformed, the table they land... - [Detections on a source](https://dfe.docs.hyperi.io/latest/dfe-engine/data-plane/source-detections.md): A source owns its detections the same way it owns its schema: the SQL runs against that source's table, so it sees exactly the columns the source d... - [The source flow - one definition, two transports](https://dfe.docs.hyperi.io/latest/dfe-engine/data-plane/source-flow.md): A source names where its records come in, whether they are transformed, and where they land. Everything else - topics, routing rules, endpoints, th... - [Storing and operating a source](https://dfe.docs.hyperi.io/latest/dfe-engine/data-plane/source-registry.md): Where a source definition lives, what its state means for the table and the deployed apps, and the surface that changes it. The file itself is sour... - [What a source compiles into](https://dfe.docs.hyperi.io/latest/dfe-engine/data-plane/source-routing.md): The engine emits the receiver's and the loader's own native config -- not a DFE wrapper the services then translate. This is what to read when a de... - [Source - the top-level data abstraction](https://dfe.docs.hyperi.io/latest/dfe-engine/data-plane/source.md): **Status:** Shipped (`SourceRegistry` + the Source model are the live path). - [The topic contract](https://dfe.docs.hyperi.io/latest/dfe-engine/data-plane/topic-contract.md): A defined source implies Kafka topics: `_land` unless it lands on the shared topic, and `_load` when the source has a transform. Th... - [Deployment](https://dfe.docs.hyperi.io/latest/dfe-engine/deployment.md): How DFE gets onto a cluster and how the engine's dials reach it. The engine never deploys backing services and never authors Argo Applications - it... - [DFE Backing Services - swap every backend by config](https://dfe.docs.hyperi.io/latest/dfe-engine/deployment/backing-services.md): DFE is a generic product that ships into clusters we never touch. So every backing service it stands on - the secrets store, the streaming bus, the... - [KEDA scaling patterns](https://dfe.docs.hyperi.io/latest/dfe-engine/deployment/keda-scaling.md): Every KEDA-scaled app scales on CPU. The ingest apps also scale on scalo ScalingPressure by default, read through the fail-safe `dfe-keda-shim`. On... - [DFE Observability Standard](https://dfe.docs.hyperi.io/latest/dfe-engine/deployment/observability-standard.md): One telemetry contract for the whole deployment: the dfe-infra platform services, dfe-engine, dfe-ui and the Rust apps (receiver, loader, fetcher, ... - [Transports and profiles](https://dfe.docs.hyperi.io/latest/dfe-engine/deployment/transports.md): DFE moves records between stages on one of two transports: a data bus, or direct gRPC. The profile picks which one, and every stage is bound to it ... - [Reference](https://dfe.docs.hyperi.io/latest/dfe-engine/reference.md): Lookup tables for the engine's HTTP API. - [dfe CLI reference](https://dfe.docs.hyperi.io/latest/dfe-engine/reference/cli.md): Rendered from the `dfe` command tree of `ghcr.io/hyperi-io/dfe-engine:v1.22.90`. - [dfe-fetcher documentation](https://dfe.docs.hyperi.io/latest/dfe-fetcher.md): How dfe-fetcher is built, and how to set up each cloud source it pulls from. - [dfe-fetcher architecture](https://dfe.docs.hyperi.io/latest/dfe-fetcher/architecture.md): The repo is a Cargo workspace: one app crate and a source framework split into an I/O-free core plus one leaf crate per shape family, with the file... - [Cloud and SaaS Setup Guides](https://dfe.docs.hyperi.io/latest/dfe-fetcher/cloud-setup.md): Each guide here covers what a provider administrator configures so dfe-fetcher can pull data from that provider: the access it needs, how to grant ... - [AWS Setup for dfe-fetcher](https://dfe.docs.hyperi.io/latest/dfe-fetcher/cloud-setup/aws.md): What a cloud administrator configures so dfe-fetcher can pull security and operational data from AWS. - [Azure Setup for dfe-fetcher](https://dfe.docs.hyperi.io/latest/dfe-fetcher/cloud-setup/azure.md): What a cloud administrator configures so dfe-fetcher can pull security and operational data from Azure and Microsoft Entra ID. - [Bitwarden Setup for dfe-fetcher](https://dfe.docs.hyperi.io/latest/dfe-fetcher/cloud-setup/bitwarden.md): What a Bitwarden organization owner needs to configure so dfe-fetcher can read organization event logs from the Bitwarden Public API. - [Cloudflare Setup for dfe-fetcher](https://dfe.docs.hyperi.io/latest/dfe-fetcher/cloud-setup/cloudflare.md): What a Cloudflare administrator needs to configure so dfe-fetcher can read account-level audit logs. - [CrowdStrike Setup for dfe-fetcher](https://dfe.docs.hyperi.io/latest/dfe-fetcher/cloud-setup/crowdstrike.md): What a CrowdStrike Falcon administrator needs to configure so dfe-fetcher can read alert data from a Falcon tenant. - [Datadog Setup for dfe-fetcher](https://dfe.docs.hyperi.io/latest/dfe-fetcher/cloud-setup/datadog.md): What a Datadog administrator configures so dfe-fetcher can read an organisation's audit trail and security signals. - [Duo Setup for dfe-fetcher](https://dfe.docs.hyperi.io/latest/dfe-fetcher/cloud-setup/duo.md): What a Cisco Duo administrator needs to configure so dfe-fetcher can read authentication logs from a Duo tenant via the Admin API. - [GCP Pub/Sub Setup for dfe-fetcher](https://dfe.docs.hyperi.io/latest/dfe-fetcher/cloud-setup/gcp-pubsub.md): What a GCP administrator needs to configure so dfe-fetcher can pull audit and log entries delivered through a Cloud Logging sink into a Pub/Sub topic. - [GCP Setup for dfe-fetcher](https://dfe.docs.hyperi.io/latest/dfe-fetcher/cloud-setup/gcp.md): What a cloud administrator configures so dfe-fetcher can read security and audit data from Google Cloud Platform (read-only, pull-mode). - [GitHub Setup for dfe-fetcher](https://dfe.docs.hyperi.io/latest/dfe-fetcher/cloud-setup/github.md): What a GitHub administrator needs to configure so dfe-fetcher can read audit-log data from a GitHub organisation or enterprise account. - [Google Workspace Setup for dfe-fetcher](https://dfe.docs.hyperi.io/latest/dfe-fetcher/cloud-setup/google-workspace.md): What a Google Workspace / GCP administrator needs to configure so dfe-fetcher can read audit and activity data from the Workspace Reports API. - [M365 Setup for dfe-fetcher](https://dfe.docs.hyperi.io/latest/dfe-fetcher/cloud-setup/m365.md): What a cloud administrator configures so dfe-fetcher can read security and audit data from Microsoft 365 (read-only, pull-mode). - [Object Store Setup for dfe-fetcher](https://dfe.docs.hyperi.io/latest/dfe-fetcher/cloud-setup/object-store.md): What a cloud administrator needs to configure so dfe-fetcher can read log objects (gzipped JSON-lines and similar) that cloud services drop into a ... - [Okta Setup for dfe-fetcher](https://dfe.docs.hyperi.io/latest/dfe-fetcher/cloud-setup/okta.md): What an Okta administrator needs to configure so dfe-fetcher can read System Log events from an Okta org. - [1Password Setup for dfe-fetcher](https://dfe.docs.hyperi.io/latest/dfe-fetcher/cloud-setup/onepassword.md): What a 1Password account owner or administrator needs to configure so dfe-fetcher can read activity from the 1Password Events Reporting API. - [runZero Setup for dfe-fetcher](https://dfe.docs.hyperi.io/latest/dfe-fetcher/cloud-setup/runzero.md): What a runZero administrator configures so dfe-fetcher can pull an organisation's asset inventory from a runZero console (read-only, pull-mode). - [Salesforce Setup for dfe-fetcher](https://dfe.docs.hyperi.io/latest/dfe-fetcher/cloud-setup/salesforce.md): What a Salesforce administrator configures so dfe-fetcher can read security and audit data from a Salesforce org (read-only, pull-mode). - [Slack Setup for dfe-fetcher](https://dfe.docs.hyperi.io/latest/dfe-fetcher/cloud-setup/slack.md): What a Slack Enterprise Grid Organization Owner needs to configure so dfe-fetcher can read audit-log events from the Slack Audit Logs API. - [Supply-Chain Registry Setup for dfe-fetcher](https://dfe.docs.hyperi.io/latest/dfe-fetcher/cloud-setup/supply-chain.md): What an operator needs to configure so dfe-fetcher can watch public package registries (PyPI, crates.io, Go module proxy) for supply-chain takeover... - [dfe-fetcher Design Document](https://dfe.docs.hyperi.io/latest/dfe-fetcher/internals.md): dfe-fetcher pulls security and operational data from cloud and SaaS providers, databases, files and external extractors, and delivers each record t... - [Reference](https://dfe.docs.hyperi.io/latest/dfe-fetcher/reference.md): Lookup pages for the database drivers, the REST profile grammar and the snapshot envelope. - [Database drivers](https://dfe.docs.hyperi.io/latest/dfe-fetcher/reference/db-drivers.md): A `sources.db` instance names an engine, and the engine decides what has to be in the image: `odbc` speaks to any SQL engine through unixODBC and t... - [The REST profile grammar](https://dfe.docs.hyperi.io/latest/dfe-fetcher/reference/profile-grammar.md): A profile is the shape of an API and carries no identity: a base URL template, the auth modes the API accepts, headers, the retry policy, where err... - [The snapshot envelope](https://dfe.docs.hyperi.io/latest/dfe-fetcher/reference/snapshot-envelope.md): A dump unit's rows travel in an envelope so a consumer can rebuild the whole store by `snapshot_id` and tell a truncated dump from a complete one. ... - [dfe-infra documentation](https://dfe.docs.hyperi.io/latest/dfe-infra.md): Start here to deploy, test and secure the DFE suite with dfe-infra. It covers the architecture, the deployment guides under `deployment/`, the vali... - [dfe-infra architecture](https://dfe.docs.hyperi.io/latest/dfe-infra/architecture.md): dfe-infra is how DFE gets onto a bare Kubernetes cluster: the Helm charts, Argo CD ApplicationSets and bootstrap. Every deployment starts from its ... - [Autoscaling: the two layers and the per-target fork](https://dfe.docs.hyperi.io/latest/dfe-infra/autoscaling.md): Two independent layers scale a DFE deployment. Keep them apart: - [ClickHouse identity](https://dfe.docs.hyperi.io/latest/dfe-infra/clickhouse-identity.md): The deploy layer creates ONE credential, and supplies one password per app. dfe-engine creates every user, role, grant, quota and row policy. - [ClickHouse native JSON type](https://dfe.docs.hyperi.io/latest/dfe-infra/clickhouse-json-type.md): > **Coming from Splunk or Elastic?** The `_json` columns in DFE's tables (`dfe.main._json`, for example) are this native type. A path nobody has ma... - [DFE cryptographic standard](https://dfe.docs.hyperi.io/latest/dfe-infra/cnsa-standard.md): DFE targets the NSA Commercial National Security Algorithm (CNSA) family, because national-security deployment is in scope. It also has to deploy i... - [Trusting the DFE certificate](https://dfe.docs.hyperi.io/latest/dfe-infra/deploy-tls-trust.md): One of two CAs signs the gateway's `*.` certificate, chosen per deployment, and the modes ask different things of a developer box. One comm... - [Deploying DFE with dfe-infra](https://dfe.docs.hyperi.io/latest/dfe-infra/deployment.md): You hand it a bare cluster and get a running DFE back. Bootstrap lays the Layer 0 baseline, then DFE's own Argo CD reconciles the backing services ... - [DFE on AWS -- configuration](https://dfe.docs.hyperi.io/latest/dfe-infra/deployment/aws-configuration.md): What the AWS root builds, how the dial sizes it, and the Kafka and ClickHouse choices behind it. The deploy steps are in aws.md. - [DFE on AWS -- Operations](https://dfe.docs.hyperi.io/latest/dfe-infra/deployment/aws-operations.md): Provisioning (the dial, sizing, Kafka and ClickHouse choices) is in aws.md. Lifecycle tags and teardown are in aws-teardown.md. Stack, re-size and ... - [DFE on AWS -- Teardown](https://dfe.docs.hyperi.io/latest/dfe-infra/deployment/aws-teardown.md): Provisioning is in aws.md. Running the cluster is in aws-operations.md. - [DFE on AWS](https://dfe.docs.hyperi.io/latest/dfe-infra/deployment/aws.md): DFE on Amazon EKS, in your own AWS account. OpenTofu builds the network, a Graviton cluster, managed Kafka and the stores. Then the same bootstrap ... - [ClickHouse SSD tiering](https://dfe.docs.hyperi.io/latest/dfe-infra/deployment/clickhouse-tiering.md): > **Coming from Splunk or Elastic?** Tiering is a ClickHouse storage policy, not table DDL. `moveFactor` triggers demotion when the hot volume's fr... - [ClickHouse: targets, operators, and how to swap them](https://dfe.docs.hyperi.io/latest/dfe-infra/deployment/clickhouse.md): ClickHouse is the DFE data layer. Which ClickHouse, and which operator if any, is a per-deployment choice behind the `clickhouse-cluster` chart's... - [Default composition: which apps a profile deploys](https://dfe.docs.hyperi.io/latest/dfe-infra/deployment/composition.md): `apps.yaml` is the source of truth. Each app carries `profiles` (where it may run) and `default_in` (where it runs when nobody says otherwise). No ... - [Edge-fleet VPN](https://dfe.docs.hyperi.io/latest/dfe-infra/deployment/edge-vpn.md): Appliances in the field have to post to the receiver. Most sit behind carrier NAT with no fixed address and nothing to open a port on. So the edge ... - [The edge module](https://dfe.docs.hyperi.io/latest/dfe-infra/deployment/edge.md): The edge module decides whether traffic from outside reaches a DFE workload. It's the public gateway (`helm/edge/gateway`) plus the fleet tunnel (c... - [Edge OIDC through the gateway](https://dfe.docs.hyperi.io/latest/dfe-infra/deployment/gateway-oidc.md): DFE has two ways to authenticate a person. Local auth is the default and needs nothing: dfe-engine holds the accounts and signs its own tokens. The... - [Kafka providers](https://dfe.docs.hyperi.io/latest/dfe-infra/deployment/kafka.md): DFE talks to Kafka through one open provider abstraction (scalo `transport/kafka/providers.rs`). Each provider handles its own auth and capabilitie... - [Kafka broker autoscaling](https://dfe.docs.hyperi.io/latest/dfe-infra/deployment/kafka/autoscaling.md): `kafka.autoscaling` drives a KEDA `ScaledObject` that scales the broker `KafkaNodePool`'s replica count directly. Strimzi 1.2.0 put a `/scale` subr... - [Running DFE on AWS MSK](https://dfe.docs.hyperi.io/latest/dfe-infra/deployment/kafka/aws-msk-operations.md): Module and inputs (provisioning, OpenTofu, wiring): aws-msk.md. - [DFE on AWS MSK](https://dfe.docs.hyperi.io/latest/dfe-infra/deployment/kafka/aws-msk.md): Operations (bootstrap Job, autoscaling, telemetry) are in aws-msk-operations.md. - [DFE on Confluent Cloud](https://dfe.docs.hyperi.io/latest/dfe-infra/deployment/kafka/confluent-cloud.md): DFE doesn't deploy Confluent Cloud. You provision it, then point DFE at it with `kafka.mode=external`. Confluent Cloud is the DFE provider... - [DFE on Redpanda Cloud](https://dfe.docs.hyperi.io/latest/dfe-infra/deployment/kafka/redpanda-cloud.md): DFE doesn't deploy Redpanda Cloud. You provision it, then point DFE at it with `kafka.mode=external`. Redpanda Cloud is the DFE provider... - [Redpanda in your cluster](https://dfe.docs.hyperi.io/latest/dfe-infra/deployment/kafka/redpanda-limitations.md): Redpanda is a Kafka-API-compatible broker that DFE can run in your cluster in place of Strimzi. It has no JVM and no separate controller nodes, whi... - [Managed backends on the deployment's own cloud](https://dfe.docs.hyperi.io/latest/dfe-infra/deployment/managed-backends.md): A managed backend is offered only as its instance on the cloud the deployment runs on. An AWS deployment gets MSK, Confluent Cloud on AWS, Redpanda... - [Migrating from DFE 2.x before 2.2](https://dfe.docs.hyperi.io/latest/dfe-infra/deployment/migrating-from-2.1.md): A private-cloud DFE deployment run before 2.2 used an internal ClickHouse fork with its own SSD cache in front of bulk storage, under an internal o... - [Node autoscaling for on-premises clusters](https://dfe.docs.hyperi.io/latest/dfe-infra/deployment/node-autoscaling.md): KEDA adds pods, and a pod with no room Pends until something adds a node. Autoscaling sets out the two layers and the per-target choice. On AWS the... - [DFE on Kubernetes on-premises](https://dfe.docs.hyperi.io/latest/dfe-infra/deployment/on-prem.md): DFE on a Kubernetes cluster you run. You start with a bare cluster and end with a stack that has sent its first event through to ClickHouse. RKE2 i... - [Deployment portability matrix](https://dfe.docs.hyperi.io/latest/dfe-infra/deployment/portability-matrix.md): DFE is one product deployed into many estates. Most components sit behind a config dial, so a deployment swaps them without a code change. A dial i... - [RKE2 - a primer (the DFE reference distribution)](https://dfe.docs.hyperi.io/latest/dfe-infra/deployment/rke2.md): DFE targets any CNCF-conformant Kubernetes. RKE2 is the reference distribution, and the one we provision when a customer has no cluster (on-prem, b... - [Stack versioning](https://dfe.docs.hyperi.io/latest/dfe-infra/deployment/stack-versioning.md): DFE is made of independently released parts: the Rust data plane, dfe-engine, dfe-ui, dfe-hyperdx, the schemas, the charts and operators, and the d... - [State, storage and repositories](https://dfe.docs.hyperi.io/latest/dfe-infra/deployment/state-and-repos.md): > **Coming from Splunk or Elastic?** DFE stores rule definitions in the deploy repo's `config/` directory, where git records their changes. Review ... - [Storage: the deploy-time matrix](https://dfe.docs.hyperi.io/latest/dfe-infra/deployment/storage.md): Where DFE's data physically lives isn't one choice but several independent ones. Look up the service, deployment mode and storage model below. Each... - [Troubleshooting toolbox](https://dfe.docs.hyperi.io/latest/dfe-infra/deployment/toolbox.md): Sometimes you need a shell inside the cluster's own network. To curl a Service by its internal DNS name, run a query against ClickHouse without a p... - [Rollback runbook](https://dfe.docs.hyperi.io/latest/dfe-infra/deployment/upgrade-rollback.md): Most of an upgrade rolls back by moving the pin, and a few steps are one way (upgrades.md, "What cannot be rolled back"). Check the stage notes bel... - [Upgrades](https://dfe.docs.hyperi.io/latest/dfe-infra/deployment/upgrades.md): Every upgrade is a committed diff in the deployment repo, applied in a declared order, with a pre-flight that names what can't be rolled back. Movi... - [The deployment wizard -- `dfe-ops init`](https://dfe.docs.hyperi.io/latest/dfe-infra/deployment/wizard.md): `dfe-ops init` walks you through the fields in `deployment.example.yaml` and writes a `deployment.yaml` dial. Defaults come from that file. Every n... - [Design principles](https://dfe.docs.hyperi.io/latest/dfe-infra/design-principles.md): These are the rules every DFE app follows. Short version here, with a link to the long version where there is one. - [Edge exposure and authentication -- cloud specifics](https://dfe.docs.hyperi.io/latest/dfe-infra/edge-auth-cloud.md): On a cloud deploy the Gateway Service is internet-facing, and public web access is default-deny. The model, the identity plane and on-prem behaviou... - [Edge exposure and authentication](https://dfe.docs.hyperi.io/latest/dfe-infra/edge-auth.md): Internet-facing exposure on a cloud deploy (CIDR, rate limit, WAF, DNS): edge-auth-cloud.md. - [The ingest edge](https://dfe.docs.hyperi.io/latest/dfe-infra/ingest-edge.md): `dfe-receiver` takes traffic from outside the cluster and produces to local Kafka, which never faces the network. The web UIs are a separate door w... - [Observability](https://dfe.docs.hyperi.io/latest/dfe-infra/observability.md): How a DFE deployment watches itself. The stack's own OTLP lands in dfe-hyperdx on every deployment. - [Provider logs in dfe-hyperdx](https://dfe.docs.hyperi.io/latest/dfe-infra/observability/provider-logs.md): Self-monitoring has two feeds: the stack's own OTLP, and the infra provider's own logs (the EKS audit log, CloudTrail, a Rancher API server audit l... - [DFE exposure model](https://dfe.docs.hyperi.io/latest/dfe-infra/threat-model.md): **Most of this stack has no internet-facing and no user-facing exposure.** Three components do, and a fourth where a deployment turns the edge tunn... - [dfe-loader docs](https://dfe.docs.hyperi.io/latest/dfe-loader.md): High-performance loader from message transports (Kafka, gRPC, Memory) into ClickHouse. Point it at a topic, give it a routing rule, and it parses, ... - [Architecture](https://dfe.docs.hyperi.io/latest/dfe-loader/architecture.md): dfe-loader is a per-message hot path with a per-table batching tail. A message arrives as bytes, gets routed and promoted into a typed row, accumul... - [DDL field-mapping directives](https://dfe.docs.hyperi.io/latest/dfe-loader/clickhouse/ddl-directives.md): dfe-loader reads the schema from ClickHouse, not from a config file -- but a plain `CREATE TABLE` does not say where each column's value comes from... - [TLS and private CAs](https://dfe.docs.hyperi.io/latest/dfe-loader/clickhouse/tls.md): With TLS on the loader speaks HTTPS (`clickhouse.protocol: http`, usually 8443) or native TLS (`native`, usually 9440), and either verifies the ser... - [ClickHouse data types](https://dfe.docs.hyperi.io/latest/dfe-loader/clickhouse/types.md): This is the type catalogue the dynamic encoder works against. When dfe-loader reflects a table from `system.columns`, every column comes back as on... - [Configuration](https://dfe.docs.hyperi.io/latest/dfe-loader/configuration.md): dfe-loader reads its config in four layers. Higher layers win. Every setting has a safe default, so a loader started with no config file boots. - [Operations](https://dfe.docs.hyperi.io/latest/dfe-loader/operations.md): How to run dfe-loader, check a config before it boots, watch it, and reason about what it does when something goes wrong. - [Capture modes](https://dfe.docs.hyperi.io/latest/dfe-loader/pipeline/capture-modes.md): `capture_mode` decides what is stored alongside the promoted columns: the full payload as `_json` (native JSON type, path-queryable), the raw bytes... - [Routing](https://dfe.docs.hyperi.io/latest/dfe-loader/pipeline/routing.md): Routing picks the `db.table` a message lands in, before any flattening, from fields in the message itself. The first hop (dfe-receiver or dfe-fetch... - [Common header schema v2](https://dfe.docs.hyperi.io/latest/dfe-loader/transport/common-header.md): Every event the loader writes -- regardless of the transport it arrived on (a Kafka topic or a gRPC push) -- gets a standardised common header inje... - [dfe-receiver documentation](https://dfe.docs.hyperi.io/latest/dfe-receiver.md): How dfe-receiver is built, hardened and tuned, and how to point a caller at it. - [dfe-receiver architecture](https://dfe.docs.hyperi.io/latest/dfe-receiver/architecture.md): Why the service is shaped the way it is, and the rules you cannot read off the code. The configuration reference is config.example.yaml. The compon... - [Internet-Facing Hardening & Fronting Architecture](https://dfe.docs.hyperi.io/latest/dfe-receiver/hardening.md): March 2026 - [dfe-receiver Design Document](https://dfe.docs.hyperi.io/latest/dfe-receiver/internals.md): dfe-receiver is a high-performance HTTP/gRPC receiver for PB/s scale data ingestion. It replaces a Vector-based implementation with native Rust for... - [Setting up an inbound webhook caller](https://dfe.docs.hyperi.io/latest/dfe-receiver/webhook-setup.md): `POST /webhook/{caller}` accepts events pushed by a product -- an alert rule, a SaaS notification hook -- rather than streamed by an agent. This pa... - [dfe-schemas documentation](https://dfe.docs.hyperi.io/latest/dfe-schemas.md): How the DFE schemas are defined, applied and kept compatible across ClickHouse platforms. - [ClickHouse platforms: one probe, one DDL](https://dfe.docs.hyperi.io/latest/dfe-schemas/clickhouse-platform.md): Every DFE table is created with the same DDL on every ClickHouse. What differs between ClickHouse Cloud and self-hosted ClickHouse is detected from... - [Compatibility profiles](https://dfe.docs.hyperi.io/latest/dfe-schemas/compat-profiles.md): A compatibility profile reads the header fields another platform, or an older DFE, writes, and sets the DFE header from them. It is data: versioned... - [The apply manifest (`manifest.yaml`)](https://dfe.docs.hyperi.io/latest/dfe-schemas/manifest.md): One list of every object a DFE deployment creates, in the order it creates them. dfe-engine iterates it at boot and applies nothing that is not in it. - [Meta Schema System](https://dfe.docs.hyperi.io/latest/dfe-schemas/meta-schema.md): A **meta schema** is a YAML file that defines the columns for a ClickHouse table - the single source of intent for what a table should look like. T... - [dfe-schemas](https://dfe.docs.hyperi.io/latest/dfe-schemas/pypi.md): The one place a DFE deployment's ClickHouse databases, tables, views, roles and grants and its Kafka bootstrap topics are defined. dfe-engine reads... - [Table definitions (`tables/`)](https://dfe.docs.hyperi.io/latest/dfe-schemas/tables.md): `tables/` holds tables written directly in ClickHouse types, as opposed to the `meta/` schemas that map DFE's 13 primitives onto ClickHouse. Five s... - [docs](https://dfe.docs.hyperi.io/latest/dfe-transform-elastic.md): Everything deeper than the README. Start at architecture.md if you are new to the service, and at compat.md if you are working a source towards par... - [Configuration](https://dfe.docs.hyperi.io/latest/dfe-transform-elastic/configuration.md): Where the values come from, which env spelling reaches a `--config` file, how the transport is selected, and which sections the file cannot carry. ... - [Enrichment](https://dfe.docs.hyperi.io/latest/dfe-transform-elastic/enrichment.md): GeoIP, user agent and community ID: the three runtime enrichers loaded once and shared across every transform. The code map around them is architec... - [Parity](https://dfe.docs.hyperi.io/latest/dfe-transform-elastic/parity.md): What this service promises against Elastic's own output, what it deliberately leaves alone, and how the claim is measured. The code map it sits bes... - [Parsers and processors](https://dfe.docs.hyperi.io/latest/dfe-transform-elastic/parsers.md): The three-layer parser strategy that replaces grok and regex, every ingest processor the transforms use, and how far the Painless matchers reach. T... - [dfe-transform-vector documentation](https://dfe.docs.hyperi.io/latest/dfe-transform-vector.md): How dfe-transform-vector is built and configured, and how to move a pipeline onto it from Vector. - [Architecture](https://dfe.docs.hyperi.io/latest/dfe-transform-vector/architecture.md): Why this repo exists, what owns what, and the invariants you cannot read off a single file. For the field-by-field design -- every config key, ever... - [dfe-transform-vector internals](https://dfe.docs.hyperi.io/latest/dfe-transform-vector/internals.md): dfe-transform-vector wraps Vector.dev as a managed subprocess to provide **Kafka -> transform -> Kafka** pipelines that the DFE platform manages, d... - [librdkafka Configuration Reference](https://dfe.docs.hyperi.io/latest/dfe-transform-vector/librdkafka.md): DFE services use librdkafka (via Vector's Kafka source/sink or via rdkafka crate directly). This document covers the shared baseline defaults, per-... - [Migration Guide: Vector → dfe-transform-vector](https://dfe.docs.hyperi.io/latest/dfe-transform-vector/migration.md): This guide covers migrating existing Vector pipelines from the official `timberio/vector` Helm chart to the `dfe-transform-vector` wrapper. - [dfe-transform-vrl documentation](https://dfe.docs.hyperi.io/latest/dfe-transform-vrl.md): The architecture and design of dfe-transform-vrl. - [Architecture](https://dfe.docs.hyperi.io/latest/dfe-transform-vrl/architecture.md): Reshaping events with Vector costs memory DFE cannot afford at volume. - [dfe-transform-vrl — Design Document](https://dfe.docs.hyperi.io/latest/dfe-transform-vrl/internals.md): `dfe-transform-vrl` is an embedded VRL transform engine for Kafka-to-Kafka pipelines. Unlike `dfe-transform-vector` (which runs Vector as a subproc... - [dfe-ui documentation](https://dfe.docs.hyperi.io/latest/dfe-ui.md): How the DFE console is built, how it talks to the engine, and how to run it locally. - [dfe-ui architecture](https://dfe.docs.hyperi.io/latest/dfe-ui/architecture.md): dfe-ui is the operator surface of the DFE suite: a Next.js app that talks to exactly one backend - the dfe-engine API - and embeds dfe-hyperdx for ... - [Authentication](https://dfe.docs.hyperi.io/latest/dfe-ui/authentication.md): Two ways to sign in, both wired through NextAuth. The UI never runs its own identity provider handshake - identity always comes from the engine. - [How the console maps to the flow](https://dfe.docs.hyperi.io/latest/dfe-ui/console-and-the-flow.md): A record enters DFE, is optionally reshaped, and lands in a table. The console is arranged in that order, so a new operator can read the menu top t... - [dfe-hyperdx documentation](https://dfe.docs.hyperi.io/latest/dfe-ui/dfe-hyperdx.md): DFE's fork of HyperDX, embedded in the DFE platform as its visualisation and search layer. - [Alert webhook template variables](https://dfe.docs.hyperi.io/latest/dfe-ui/dfe-hyperdx/alert-webhook-template-variables.md): **Generic** and **incident.io** webhook bodies are Handlebars templates. These variables are available inside them, so a receiver can route, filter... - [OIDC authentication](https://dfe.docs.hyperi.io/latest/dfe-ui/dfe-hyperdx/architecture/oidc-authentication.md): **The DFE engine is the single JWT issuer. dfe-hyperdx verifies its token rather than trusting a header.** - [Team identity](https://dfe.docs.hyperi.io/latest/dfe-ui/dfe-hyperdx/architecture/team-identity.md): **A dfe-hyperdx team is the ClickHouse user its members read as.** - [Creating a Rule from a Search](https://dfe.docs.hyperi.io/latest/dfe-ui/hyperdx-rule-create.md): DFE UI integrates with dfe-hyperdx so users can create rules directly from search results. When a user runs a search and clicks the **Create Rule**... - [The /observe embed - the Observe explore surface](https://dfe.docs.hyperi.io/latest/dfe-ui/observe-embed.md): `/observe/[[...feature]]` iframes dfe-hyperdx in chromeless mode (`?embed=1`) so the dfe-ui shell owns navigation while dfe-hyperdx provides the Ki... - [How we got here](https://dfe.docs.hyperi.io/latest/history.md): DFE 2.2 is the first open-source release of a platform in production since 2018, which is why public versions start at 2.2. - [Operate](https://dfe.docs.hyperi.io/latest/operate.md): DFE is deployed. Now it needs people, data, and a way to tell it's healthy. - [Day to day and troubleshooting](https://dfe.docs.hyperi.io/latest/operate/day-to-day.md): Start with the target you run. - [Detections and hunts](https://dfe.docs.hyperi.io/latest/operate/detections-and-hunts.md): A rule is one SQL query tied to a source. A hunt runs rules on a schedule and writes what they match to the `alerts` table. - [Monitoring](https://dfe.docs.hyperi.io/latest/operate/monitoring.md): Three questions, three places. - [Sources](https://dfe.docs.hyperi.io/latest/operate/sources.md): DFE ships a catalogue of ready-made sources. Browse it, pick the ones you run, and create them in one call instead of writing each definition by hand. - [Splunk transforms](https://dfe.docs.hyperi.io/latest/operate/splunk-transforms.md): dfe-transform-splack brings Splunk-compatible transforms to DFE. - [Transforms](https://dfe.docs.hyperi.io/latest/operate/transforms.md): A transform reshapes a source's records between the receiver and the loader. It's optional. Without one, records go straight to the loader. With on... - [Users and OIDC](https://dfe.docs.hyperi.io/latest/operate/users-and-oidc.md): Two local accounts work from the first minute, `admin` and `breakglass`. Your identity provider (IdP) comes next. Do it in this order. - [Quickstarts](https://dfe.docs.hyperi.io/latest/quickstart.md): Four ways in, each a handful of commands. Start with the demo if you have not seen DFE work yet. - [Quickstart: AWS](https://dfe.docs.hyperi.io/latest/quickstart/aws.md): OpenTofu builds Amazon EKS on Graviton, with Kafka from Amazon MSK, Confluent Cloud or Redpanda Cloud. From there, the same bootstrap as any other ... - [Quickstart: the demo](https://dfe.docs.hyperi.io/latest/quickstart/demo.md): Raw firewall syslog in, one parsed ClickHouse table per vendor out, on one host. The fetcher pulls two live API feeds beside it: Wikipedia's recent... - [Quickstart: Docker](https://dfe.docs.hyperi.io/latest/quickstart/docker.md): The whole DFE stack under Docker Compose on one host: a laptop, a demo box or a proof of concept. It runs the same images as a Kubernetes deploymen... - [Quickstart: Kubernetes on-premises](https://dfe.docs.hyperi.io/latest/quickstart/kubernetes.md): DFE on a Kubernetes cluster you run. We test on RKE2, but the charts don't need any particular distribution. ## Section bundles - [culvert](https://dfe.docs.hyperi.io/latest/llms/culvert.txt): every page in one file - [dfe-archiver](https://dfe.docs.hyperi.io/latest/llms/dfe-archiver.txt): every page in one file - [dfe-deploy](https://dfe.docs.hyperi.io/latest/llms/dfe-deploy.txt): every page in one file - [dfe-docker](https://dfe.docs.hyperi.io/latest/llms/dfe-docker.txt): every page in one file - [dfe-engine](https://dfe.docs.hyperi.io/latest/llms/dfe-engine.txt): every page in one file - [dfe-fetcher](https://dfe.docs.hyperi.io/latest/llms/dfe-fetcher.txt): every page in one file - [dfe-infra](https://dfe.docs.hyperi.io/latest/llms/dfe-infra.txt): every page in one file - [dfe-loader](https://dfe.docs.hyperi.io/latest/llms/dfe-loader.txt): every page in one file - [dfe-receiver](https://dfe.docs.hyperi.io/latest/llms/dfe-receiver.txt): every page in one file - [dfe-schemas](https://dfe.docs.hyperi.io/latest/llms/dfe-schemas.txt): every page in one file - [dfe-transform-elastic](https://dfe.docs.hyperi.io/latest/llms/dfe-transform-elastic.txt): every page in one file - [dfe-transform-vector](https://dfe.docs.hyperi.io/latest/llms/dfe-transform-vector.txt): every page in one file - [dfe-transform-vrl](https://dfe.docs.hyperi.io/latest/llms/dfe-transform-vrl.txt): every page in one file - [dfe-ui](https://dfe.docs.hyperi.io/latest/llms/dfe-ui.txt): every page in one file - [overview](https://dfe.docs.hyperi.io/latest/llms/overview.txt): every page in one file