Skip to main content

Configuration

Where the values come from, which env spelling reaches a --config file, how the transport is selected, and which sections the file cannot carry. The field-by-field reference is generated rather than written here: config-schema.yaml beside this page carries every key with its description, and config.example.yaml at the repo root is the complete default set, pinned against drift by a test. dfe-transform-elastic emit-config reprints it.

The configuration is read once, at startup​

Config::load reads the configuration once and hands it to the batch loop by reference, so every value needs a restart to change. There is no hot reload: the broker connections, the resolved transform, the GeoIP databases and the metrics labels are all fixed at startup.

The two ways in are not equivalent. With no --config the scalo cascade applies: defaults, settings and settings.<env> files, .yaml or .yml, searched in ./, ./config/, /config/ and ~/.config/dfe-transform-elastic/, then the DFE_TRANSFORM_ELASTIC_* environment. With --config, which is what the container passes, Config::load reads the named file into Config and also hands it to scalo's cascade as its settings layer, so the scalo sections written in it apply. Kafka credentials are unaffected either way: they are read from KAFKA_* separately.

Only the flat env spelling reaches a --config file​

The FLAT, single-underscore form is applied to the loaded configuration on both branches, so it works against a --config deployment:

DFE_TRANSFORM_ELASTIC_SOURCE_TOPICS=a,b
DFE_TRANSFORM_ELASTIC_SOURCE_BATCH_SIZE=5000
DFE_TRANSFORM_ELASTIC_SINK_TOPIC=out

It covers every source.* and sink.* field. geoip is not among them: it is scalo's own type, so it stays settable from the file and the cascade only. scalo's DOUBLE-underscore form is resolved from the cascade, and a --config deployment reads source.* and sink.* straight from the file, so it never reaches them.

transport selects the bus or a direct push, per side​

Each side carries a transport of bus (the default) or direct. A config file and the flat env form also take kafka for bus and grpc for direct, the names dfe-engine renders, and a config always writes bus and direct back. A config file refuses any other name, and an unrecognised env value keeps the configured transport rather than moving the deployment. The chart publishes the push port 6000 when source.transport is direct or grpc. Both transports are compiled in by default and both are constructed: on direct the service binds a scalo Push listener on source.listen (default 0.0.0.0:6000) and pushes to sink.endpoint (default http://dfe-loader:6000). The listener answers a push only once its events are delivered, so a refused push is the upstream sender's to retry.

DFE_TRANSFORM_ELASTIC_SOURCE_TRANSPORT=direct
DFE_TRANSFORM_ELASTIC_SOURCE_LISTEN=0.0.0.0:6000
DFE_TRANSFORM_ELASTIC_SINK_ENDPOINT=http://dfe-loader:6000

Every source and sink key may be left out of the file. source.name, source.topics, source.group_id, source.brokers and sink.topic then default to empty. An instance with no source name idles, as does a bus instance with no topics. Once the configuration names work, startup refuses a bus source with no brokers or group.

On direct the brokers and group may stay empty. Empty topics do not idle there, because the listener is the work. In a DFE deployment the slim and mesh profiles run both sides on direct, and single and scale run on bus.

sink.topic is required on both transports once the configuration names work. On direct it is the routing key every push carries, and dfe-loader routes a record with no _source on that key, so a push without one lands under the loader's default_topic.

The listener speaks plaintext gRPC with no authentication of its own, and the push to the endpoint travels in the clear. Both belong behind the mesh route dfe-infra provisions for them, never on an interface reachable from outside the cluster.

Agentless mode serves one syslog stream of many device types​

source.mode: agentless reads the source off each event instead of from source.name, so one pod can consume a dfe-receiver syslog topic that carries firewalls, switches and proxies together. Each event is matched against a table of markers (literals, key sets and CEF vendor and product pairs), the winner's transform runs, and the output is stamped _source with that source's dfe-schemas catalogue label, which is what dfe-loader routes on. The markers are catalogue data: each source's catalogue.detect block in dfe-schemas, compiled into src/detect/markers.rs from the release scripts/dfe-schemas.toml pins. source.name must stay empty. source.envelope must be auto or receiver.

  • source.mode: agent (default) or agentless.

  • source.detect.unmatched: what an event no marker claims gets. passthrough (default) unwraps it to the Beats layout, runs no transform and leaves _source off, so the loader lands it under sink.topic's default table. fallback runs source.detect.fallback's transform and stamps its label. stop ends the service with the block unreleased, so a restart reads it again.

  • source.detect.fallback: a registered source, required when unmatched is fallback and refused otherwise.

  • source.detect.candidates: the sources the detector may name. Empty means all of them. A name it does not know is refused at startup. Narrowing it hands shared lines to the source left in: ASA-format lines go to ASA, and to the IOS catch-all once ASA is out. Citrix CEF goes to the ADC source unless it is excluded, and then to WAF.

    DFE_TRANSFORM_ELASTIC_SOURCE_MODE=agentless DFE_TRANSFORM_ELASTIC_SOURCE_DETECT_UNMATCHED=fallback DFE_TRANSFORM_ELASTIC_SOURCE_DETECT_FALLBACK=filebeat.cisco_ios.default DFE_TRANSFORM_ELASTIC_SOURCE_DETECT_CANDIDATES=filebeat.cisco_asa.default,filebeat.cisco_ftd.default

Fetcher events carry the provider's own payload, so they count as unmatched. Three counters report the split: events_detected_total{source}, events_unmatched_total and events_fallback_total.

Detection costs about 2,080 ns an event (481K events a second on one core), so a 20,000-event batch spends about 42 ms on it. That is 20,000 compat-corpus events of 718 bytes on average, measured on one AMD EPYC 7302 core with the bench profile. cargo bench --bench detect re-derives it.

Run it on the receiver's _raw capture where you can. The receiver splits each line with syslog_loose, and a marker that straddled the split (Pleasant Password Server:, an amp: tag) is gone from the parsed fields. On the compat corpus the detector names 8,243 of 8,885 events of detectable sources from the raw line and 8,004 from the parsed fields.

scripts/replay_syslog.py feeds an agentless stack from the committed fixtures under tests/fixtures/unencumbered/, standard library only. --sources takes catalogue labels from the same table the detector stamps, and --list shows the ones with fixtures. A syslog header is added only to a fixture line that lacks one, and the % the ASA fixtures dropped is put back. UDP sends one datagram per line and TCP is newline framed. There is no http:// form, because dfe-receiver's /ingest takes JSON and not syslog.

scripts/replay_syslog.py --list
scripts/replay_syslog.py --target udp://dfe-receiver:514 --sources cisco-asa,fortinet,panw --rate 50 --loop --duration 120

source.acknowledgements holds the source until delivery​

source.acknowledgements.enabled (default true) holds the source's acknowledgement -- the Kafka offset commit on the bus, the answer to a push on direct -- until every event built from a record is delivered, or dropped and counted. A sink outage is then waited out, and a crash redelivers rather than loses. false acknowledges at receipt, before the transform runs, and a crash loses what was in flight. The flat env form is DFE_TRANSFORM_ELASTIC_SOURCE_ACKNOWLEDGEMENTS_ENABLED.

On a direct source a push is held at most 18 s, so every send it waits on gives up at 15 s: a direct sink's gRPC deadline, and a bus sink's message.timeout.ms (librdkafka's default is 300 s).

sink.max_message_bytes caps one outbound record​

The default is the DFE record cap, 8,000,000 bytes: dfe-receiver refuses a request body over it, and every broker DFE deploys takes a record that size (Confluent Basic's 8,388,608 limit leaves room for the framing). The producer this service builds allows 16 MiB, so a deployment with larger brokers raises the budget up to that. An event that serialises larger is dropped, logged, counted on events_oversize_total, and its source is still acknowledged.

Against a broker or topic whose message.max.bytes is lower -- a stock Kafka broker takes about 1 MB -- set sink.max_message_bytes below that limit. Left at the default, a broker below 8,000,000 bytes refuses the events between the two instead: each is dropped as a dead letter, counted on send_filtered_dlq_total, and the partition moves on. Validation refuses a value under 4096 or over 16,777,216.

DFE_TRANSFORM_ELASTIC_SINK_MAX_MESSAGE_BYTES=900000

A record that will not parse goes to the DLQ, one that will not transform is counted​

A record that is not JSON, and an event the sink transport would refuse, go to the DLQ dlq configures: by default NDJSON files under /var/spool/scalo/dlq, on the dlq volume the chart mounts at 1Gi. The source is acknowledged only once the DLQ holds them, so a full or broken DLQ holds the block rather than losing it. dlq is scalo's DlqConfig, so mode, file.rotation and file.max_age_days are set there. With dlq.enabled: false they are dropped and counted on pipeline_dead_letters_dropped_total instead.

An event whose transform returns an error, or that will not serialise, is dropped and counted on events_errored_total, and its source is acknowledged, so it is not redelivered.

scalo's own sections are set in the same file​

The --config file is a layer of scalo's cascade, so scaling, self_regulation, logger and the other sections scalo reads for itself apply when written in it. The DOUBLE-underscore env form overrides them, for example DFE_TRANSFORM_ELASTIC_SCALING__MEMORY_GATE_THRESHOLD. The effective scaling values are logged once at startup, with whether they came from the cascade or the default.

SCALO_SECTIONS in src/config/loader.rs is the full list, and the unread-key warning skips it. One of them does nothing here: batch_processing sizes scalo's own batch engine, which the service loop does not run. Its receives take their record cap from source.batch_size and their byte cap from source.max_batch_bytes. scalo still refuses to start on a batch_processing section that does not parse. Every other key no field and no scalo section reads gets one warning at startup. geoip is not on the list: the service declares that section itself and hands it to scalo. KEDA replica scaling is separate: it is driven by the chart's keda.* values and the scaling_pressure gauge.

The contract's ScaledObject scales on CPU alone (80%, 1 to 10 replicas). In a DFE deployment, dfe-infra's values set the CPU target to 70% in every profile. The single, scale and mesh profiles add the scaling-pressure trigger (target 70, read through dfe-keda-shim), and slim runs on CPU alone. The engine's KEDA scaling page covers the triggers for every app.

The metrics listener is scalo's, not a config field​

/livez, /readyz, /metrics and /metrics/manifest are served from scalo's --metrics-addr (env METRICS_ADDR, default 0.0.0.0:9090), so charts and deployments override that rather than a YAML field. On direct the push listener is a second port, 6000 by default, declared in the deployment contract so the generated Dockerfile exposes both.