Compatibility profiles

A compatibility profile reads the header fields another platform, or an older DFE, writes, and sets the DFE header from them. It is data: versioned rule rows that dfe-engine compiles into the first hop's ordinary rules, so dfe-receiver and dfe-fetcher never learn that profiles exist. Profiles live in compat-profiles/, one file per profile, and ship in the wheel under dfe_schemas/data/compat-profiles/. Two ship:

Profile	Reads	Sets	Off until a source opts in
dfe-legacy-2x	tags.event.category through the aliases table	_source
	tags.event.org_id, then org_id, skipping unknown and dfesystem	_org_id
elastic-agent	data_stream.dataset through the aliases table	_source
	data_stream.namespace	_org_id	org-namespace

DFE 2.1 wrote unknown where it had no org and dfesystem for its DLQ. Neither is a tenant, so both fall through to the default org instead of counting as a clamp violation. A 2.x receive time needs no row: the first hop stamps its own _timestamp_received, and the header reads timestamp_received and received_at after it.

data_stream.dataset is the identifier Elastic gives each integration's data stream (cisco_ios.log), so one lookup covers every Elastic-shipped source. A namespace is free text the sender picks, which is why it names a tenant only on a source that says so. Elastic's data_stream.namespace commonly defaults to default, which maps to the DFE default org when the org-namespace row is opted in.

The format

id: elastic-agent              # the file name, and what a source binds as <id>@<version>
resource_type: core            # core ships here and refuses edits, custom is an operator's own
current: "1.0.0"
versions:
  "1.0.0":
    date: "2026-10-21"
    type: model                # SchemaVer: model, addition or revision
    summary: "..."
    title: Elastic Agent       # the UI name
    description: >-            # the UI tooltip
      ...
    covers:                    # the senders it reads, shown as a chip
      - platform: elastic-agent
    targets: [_source, _org_id]
    lookups:
      aliases: {}              # empty here, filled by the engine
    rules:
      - id: dataset
        set: {_source: {lookup: aliases, from: data_stream.dataset}}
      - id: org-namespace
        match: {field: data_stream.namespace, op: exists}
        set: {_org_id: {from: data_stream.namespace}}
    opt_in: [org-namespace]

Two checks own a profile between them. compat-profiles/profile.schema.json (JSON Schema 2020-12) owns the document: the version tree, the fields a row may set, ids and the opt_in list. scalo.rules owns what a row means, so targets, lookups and rules are exactly a scalo.rules rule set, and the tests compile every version with scalo's own checker. The row format, its tests and its actions are in scalo-py's docs/api/RULES.md.

A profile sets scalar header fields only: _source, _org_id and _timestamp. Moving a subtree or reshaping a payload is a transform's job. Each version is a complete snapshot and a published one never changes, so a fix is a new version entry.

Compiling and adding one

dfe_schemas.compat.rule_set() hands one version out as the mapping scalo.rules.RuleSet.compile takes. Its name is the profile id, and a row listed under opt_in is left out unless the caller names it.

from scalo.rules import Clamp, CompileOptions, RuleSet

from dfe_schemas.compat import rule_set

config = rule_set("elastic-agent", opt_in=["org-namespace"])
config["lookups"]["aliases"] = {"cisco_ios.log": "cisco-ios"}
rules = RuleSet.compile(config, CompileOptions())
outcome = rules.evaluate(record, Clamp("_org_id", {"default", "ep"}, "default"))

A shipped aliases table is empty. dfe-engine fills it from each registered source's aliases for that profile before it compiles, so a value no source claims misses the lookup and the next row decides. A profile row that copies a sender's field into _org_id is bounded the way a sender's own _org_id is: the first hop clamps the result to the orgs the sender's credential is bound to.

To add a profile:

1. Write compat-profiles/<id>.yaml. The file name is the id: lower-case words joined by hyphens.
2. List only header fields under targets, _source before _org_id, so an org row can test the source the same pass chose.
3. Add the id to SHIPPED in tests/test_compat_profiles.py, with a test that evaluates a real record from that sender.
4. make test, then release the wheel and raise the floor in dfe-engine, as for any other definition.