DFE Expression Language — CEL

DFE uses CEL (Common Expression Language) for conditions across the Python engine, the Rust services and the CLI. CEL is not Turing-complete: no loops, no recursion, no side effects, so every expression terminates. An expression is compiled once and evaluated against every row.

Packages:

Language	Package	Version	Notes
Rust	cel (renamed from cel-interpreter)	0.14	Native implementation, through scalo::expression
Python	common-expression-language	0.10	PyO3 bindings, through scalo.expression

References:

• CEL Specification
• CEL Overview
• cel crate
• Python CEL package

---

What CEL Does NOT Replace

System	Reason
@directive: body DSL (@source, @generated, etc.)	Schema metadata — column mapping instructions, not boolean conditions
ClickHouse SQL in @generated expressions	Runs in ClickHouse, not our evaluator
Hunt SQL queries / rule_rewriter	ClickHouse-native SQL
Duration strings ("1h", "30m")	Simple config values, not expressions

---

DFE Expression Profile

DFE uses a subset of CEL. Expressions outside this profile are rejected at validation time.

Allowed Operations

Category	Operations	Examples
Comparison	==, !=, <, <=, >, >=	severity == "critical"
Logical	&&, ||, !	amount > 1000 && !is_test
Membership	in	status in ["active", "pending"]
Arithmetic	+, -, *, /, %	price * quantity
Ternary	? :	is_admin ? 95 : 50

Allowed Functions

Function	Purpose	Example
contains()	Substring check	message.contains("error")
startsWith()	Prefix check	path.startsWith("/api/")
endsWith()	Suffix check	file.endsWith(".log")
matches()	Regex match. A hunt rule's filter takes it. An alert condition takes it only when the config cascade sets expression.allow_regex: true	hostname.matches("^web-[0-9]+$")
size()	Length of string or list	size(tags) > 0
has()	Field existence	has(event.user_id)
int()	Cast to integer	int(port_str) == 443
double()	Cast to float	double(score_str) > 0.5
string()	Cast to string	string(code) == "200"
bool()	Cast to boolean	bool(enabled)

Excluded (Rejected at Validation)

Feature	Reason
map(), filter(), exists(), all()	Per-element iteration — unpredictable performance on large lists
timestamp(), duration()	ClickHouse handles time natively — no need to duplicate
Macros	Complexity with no benefit for DFE use cases

---

Supported Types

CEL Type	Python	Rust	ClickHouse
int	int	i64	Int64
double	float	f64	Float64
string	str	String	String
bool	bool	bool	Bool
list	list	Vec<Value>	Array
map	dict	HashMap<String, Value>	—
null	None	Option::None	Nullable

---

Usage Examples

Alert Triggers

alerts:
  triggers:
    - when: "result_count >= 10"

A trigger sees only result_count, the rows the fire wrote. A trigger on a row field never fires.

Source Routing

A source selects its records with when, a CEL condition, in place of match. The receiver evaluates it against every inbound record under its CEL Fast profile, so regex, iteration and time functions are refused at save. See source-definition.md.

when: "data_stream.dataset == 'cisco_ios.log' && agent.type == 'filebeat'"

Computed Values (Transforms)

columns:
  risk_label:
    expr: '@computed: risk_score > 80 ? "high" : risk_score > 40 ? "medium" : "low"'

---

CEL-to-ClickHouse SQL Transpilation

CEL expressions can be mechanically converted to ClickHouse WHERE clauses for query pushdown:

CEL	ClickHouse SQL
severity == "critical"	severity = 'critical'
amount > 10000 && !is_test	amount > 10000 AND NOT is_test
status in ["active", "pending"]	status IN ('active', 'pending')
message.contains("error")	position(message, 'error') > 0
path.startsWith("/api/")	startsWith(path, '/api/')
hostname.matches("^web-[0-9]+$")	match(hostname, '^web-[0-9]+$')
has(user_id)	user_id IS NOT NULL
size(tags) > 0	length(tags) > 0

This enables expressions written once in YAML to work both as in-memory evaluation (Rust/Python) and as ClickHouse query filters.

---

Implementation Architecture

┌─────────────────────────────────────────────┐
│              DFE Expression Profile          │
│         (subset of CEL, documented here)     │
└─────────┬───────────────────┬───────────────┘
          │                   │
    ┌─────▼──────┐    ┌──────▼──────┐
    │  scalo-py   │    │  scalo-rs    │
    │ expression/ │    │ expression/  │
    │             │    │              │
    │ wraps:      │    │ wraps:       │
    │ common-     │    │ cel          │
    │ expression- │    │ crate        │
    │ language    │    │              │
    └─────┬──────┘    └──────┬──────┘
          │                   │
    ┌─────▼──────┐    ┌──────▼──────┐
    │ dfe-engine  │    │ dfe-loader   │
    │ hunts/      │    │ transformer  │
    │ alert.py    │    │ .rs          │
    └─────────────┘    └─────────────┘

Both wrappers enforce the same DFE profile — identical allowed operators, functions, and types. The Rust crate is the single source of truth for parsing and evaluation semantics.

API Surface (Both Languages)

compile(expr) → Program          # Parse + validate + compile to AST
evaluate(expr, data) → Any       # One-shot convenience (compile + execute)
validate(expr) → list[str]       # Syntax + profile check, returns errors for UI