Skip to main content

Detections on a source

A source owns its detections the same way it owns its schema: the SQL runs against that source's table, so it sees exactly the columns the source declared. The source itself is source.md.

Rules​

A rule is one detection condition on a source's table: a SQL WHERE fragment, a CEL filter the engine transpiles to one, or both ANDed together.

# rules/brute_force_login.yaml -- the filename stem is the rule's id
rule_id: brute_force_login
name: Brute Force Login Attempt
severity: high
source_table: winaudit                  # In the data database unless source_db names another
where_clause: "event_id = 4625"

Hunts​

A hunt groups rules, names the customers whose rows they read, and carries the schedule. The runner runs rate schedules: anchored hunts are recognised and skipped.

# hunts/windows_logins.yaml -- the filename stem is the hunt's id
name: Windows logins
schedule:
  mode: rate
  interval: 5m
customers: [acme]
rules: [brute_force_login]

Matches land in <data database>.detection unless the hunt (global_target_table_name) or a rule entry (target_table_name) names another table. How the runner scales is hunt-runner-scaling.md, and how a person finds a match is hunt-match-delivery.md.

Hunt customers​

A hunt's customers are org names. The engine resolves each to the org's tenant ids when it saves the hunt and stores them as customer_tenant_ids, and the runner reads only rows whose _org_id is one of them. Every org create, update and delete stores that org's tenant ids afresh in each hunt naming it, as one deploy-repo change, before the org itself is written. A deleted org keeps its name in customers and loses its stored ids, so the hunt fails every fire as unknown_customers and holds its watermark until someone edits it. A hunt carrying a direct query instead of rules is trusted SQL from the operator-controlled deploy repo, and no customer holds it.

In production+team the hunt change waits on review while the org is written at once. So an org create or update is refused with 409 tenant_id_held when it takes a tenant id a hunt reads for another org, on main or on an unmerged review branch. Merge or delete that branch, or edit the hunt, to release the id. A squash merge leaves the branch behind, so turn on the forge's auto-delete of merged head branches. The startup seed skips such an org, and one the forge cannot check, until the next boot.

Accepted residual: the runner keeps the hunts it loaded until its next reload (--reload-every ticks of hunts.runner_poll_seconds, 20 x 15 s = 300 s at the defaults, plus huntRunner.gitSync.period on Kubernetes), so a tenant id deleted and given to a new org inside that window can fire once on the old org's hunt.

Lookup databases​

A rule condition reads its own row and one more kind of table: a table of a database hunts.lookup_databases lists (DFE_ENGINE_HUNTS_LOOKUP_DATABASES, comma-separated, empty by default). The forms are x IN (SELECT <columns> FROM <db>.<table> [WHERE <condition>]), x IN <db>.<table> and x IN (<db>.<table>), with NOT and GLOBAL as usual. Any other subquery, join, union or table after IN is refused when the rule is saved, in a governance action and when the runner compiles it.

The runner's ClickHouse user reads every org, so a listed database must hold shared reference data, such as threat-intel indicators, and no org's rows. Each of its tables must be a plain store of its own rows, on a MergeTree-family, Log, TinyLog, StripeLog, Memory or EmbeddedRocksDB engine, with no _org_id column. So a view, Merge, Distributed, Set or Join table is refused. The runner refuses to start while one is there, and stops at the reload that first finds one. The engine refuses to start too when ClickHouse answers at its start. When it does not, the engine starts with the check unrun and logs a warning, and the runner's check still applies. system, information_schema and the data database cannot be listed. The runner's hunt_runner service role needs SELECT on each listed database, added to its grants as a governed-ops edit.

Sigma views​

Sigma rules reference fields by standardised names (SourceIP, CommandLine, EventID). Rather than carrying those names in the base schema, a source declares a Sigma view and the engine generates it:

-- Auto-generated from the source schema + the sigma field mapping
CREATE VIEW {db}.{source}_sigma AS
SELECT
    source_ip AS SourceIP,
    dest_ip AS DestinationIP,
    user_name AS User,
    command_line AS CommandLine,
    event_id AS EventID,
    *
FROM {db}.{source}

The rules query the view, the base schema stays ClickHouse-native, and the mapping is per-source because different sources map differently. A view costs no storage.

# In the source definition (a views entry with standard: sigma)
views:
  - standard: sigma
    taxonomy: windows                   # Sigma logsource product binding
    custom_mappings:                    # Per-source overrides (win over field_map)
      CommandLine: command_line
      ParentCommandLine: parent_cmd

A converted Sigma rule becomes a DFE rule tied to the source, querying the {source}_sigma view. The provider pipeline that keeps the rule catalogue current is sigma.md; the mapping layer itself is field-mapping.md.