# Detections on a source

> A source owns its detections the same way it owns its schema: the SQL runs
> against that source's table, so it sees exactly the columns the source
> declared. The source itself is [source.md](/v/2.2.2/dfe-engine/data-plane/source.md).

## Rules

A rule is one detection condition on a source's table: a SQL WHERE fragment, a
CEL filter the engine transpiles to one, or both ANDed together.

```yaml
# rules/brute_force_login.yaml -- the filename stem is the rule's id
rule_id: brute_force_login
name: Brute Force Login Attempt
severity: high
source_table: winaudit                  # In the data database unless source_db names another
where_clause: "event_id = 4625"
```

## Hunts

A hunt groups rules, names the customers whose rows they read, and carries the
schedule. The runner runs `rate` schedules: `anchored` hunts are recognised and
skipped.

```yaml
# hunts/windows_logins.yaml -- the filename stem is the hunt's id
name: Windows logins
schedule:
  mode: rate
  interval: 5m
customers: [acme]
rules: [brute_force_login]
```

```mermaid
flowchart TD
    R[rule: condition on a source table] --> H[hunt runner, on the hunt's interval]
    H -->|runs each rule over the window| M[matches]
    M --> A[(data database .detection)]
```

Matches land in `<data database>.detection` unless the hunt
(`global_target_table_name`) or a rule entry (`target_table_name`) names another
table. How the runner scales is
[hunt-runner-scaling.md](/v/2.2.2/dfe-engine/data-plane/hunt-runner-scaling.md), and how a person finds a
match is [hunt-match-delivery.md](/v/2.2.2/dfe-engine/data-plane/hunt-match-delivery.md).

## Hunt customers

A hunt's `customers` are org names. The engine resolves each to the org's tenant ids when it saves the hunt and stores them as `customer_tenant_ids`, and the runner reads only rows whose `_org_id` is one of them. Every org create, update and delete stores that org's tenant ids afresh in each hunt naming it, as one deploy-repo change, before the org itself is written. A deleted org keeps its name in `customers` and loses its stored ids, so the hunt fails every fire as `unknown_customers` and holds its watermark until someone edits it. A hunt carrying a direct `query` instead of `rules` is trusted SQL from the operator-controlled deploy repo, and no customer holds it.

In production+team the hunt change waits on review while the org is written at once. So an org create or update is refused with 409 `tenant_id_held` when it takes a tenant id a hunt reads for another org, on main or on an unmerged review branch. Merge or delete that branch, or edit the hunt, to release the id. A squash merge leaves the branch behind, so turn on the forge's auto-delete of merged head branches. The startup seed skips such an org, and one the forge cannot check, until the next boot.

Accepted residual: the runner keeps the hunts it loaded until its next reload (`--reload-every` ticks of `hunts.runner_poll_seconds`, 20 x 15 s = 300 s at the defaults, plus `huntRunner.gitSync.period` on Kubernetes), so a tenant id deleted and given to a new org inside that window can fire once on the old org's hunt.

## Lookup databases

A rule condition reads its own row and one more kind of table: a table of a database `hunts.lookup_databases` lists (`DFE_ENGINE_HUNTS_LOOKUP_DATABASES`, comma-separated, empty by default). The forms are `x IN (SELECT <columns> FROM <db>.<table> [WHERE <condition>])`, `x IN <db>.<table>` and `x IN (<db>.<table>)`, with `NOT` and `GLOBAL` as usual. Any other subquery, join, union or table after `IN` is refused when the rule is saved, in a governance action and when the runner compiles it.

The runner's ClickHouse user reads every org, so a listed database must hold shared reference data, such as threat-intel indicators, and no org's rows. Each of its tables must be a plain store of its own rows, on a `MergeTree`-family, `Log`, `TinyLog`, `StripeLog`, `Memory` or `EmbeddedRocksDB` engine, with no `_org_id` column. So a view, `Merge`, `Distributed`, `Set` or `Join` table is refused. The runner refuses to start while one is there, and stops at the reload that first finds one. The engine refuses to start too when ClickHouse answers at its start. When it does not, the engine starts with the check unrun and logs a warning, and the runner's check still applies. `system`, `information_schema` and the data database cannot be listed. The runner's `hunt_runner` service role needs `SELECT` on each listed database, added to its grants as a governed-ops edit.

## Sigma views

Sigma rules reference fields by standardised names (`SourceIP`, `CommandLine`,
`EventID`). Rather than carrying those names in the base schema, a source
declares a Sigma view and the engine generates it:

```sql
-- Auto-generated from the source schema + the sigma field mapping
CREATE VIEW {db}.{source}_sigma AS
SELECT
    source_ip AS SourceIP,
    dest_ip AS DestinationIP,
    user_name AS User,
    command_line AS CommandLine,
    event_id AS EventID,
    *
FROM {db}.{source}
```

The rules query the view, the base schema stays ClickHouse-native, and the
mapping is per-source because different sources map differently. A view costs
no storage.

```yaml
# In the source definition (a views entry with standard: sigma)
views:
  - standard: sigma
    taxonomy: windows                   # Sigma logsource product binding
    custom_mappings:                    # Per-source overrides (win over field_map)
      CommandLine: command_line
      ParentCommandLine: parent_cmd
```

A converted Sigma rule becomes a DFE rule tied to the source, querying the
`{source}_sigma` view. The provider pipeline that keeps the rule catalogue
current is [sigma.md](/v/2.2.2/dfe-engine/data-plane/sigma.md); the mapping layer itself is
[field-mapping.md](/v/2.2.2/dfe-engine/data-plane/field-mapping.md).
