Migration Guide: Vector → dfe-transform-vector

This guide covers migrating existing Vector pipelines from the official timberio/vector Helm chart to the dfe-transform-vector wrapper.

Pipeline Values Migration

A values file for the official Vector chart carries the whole Vector config. dfe-transform-vector's chart takes the "big-dial" config pattern instead, one values file per pipeline. Both releases can run side by side while you move.

Values Format

Before (official Vector chart — vector-syslog.yaml):

replicas: 3
image:
  tag: "0.48.0-debian"
persistence:
  size: 20Gi
resources:
  requests:
    cpu: "2"
    memory: 2Gi
  limits:
    cpu: "4"
    memory: 4Gi
customConfig:
  sources:
    kafka_in:
      type: kafka
      bootstrap_servers: "${KAFKA_BROKERS_SASL_SCRAM}"
      group_id: dfe-vector-syslog
      topics: [raw_syslog_land]
      sasl:
        enabled: true
        mechanism: SCRAM-SHA-512
        username: "${KAFKA_SASL_USERNAME}"
        password: "${KAFKA_SASL_PASSWORD}"
      tls:
        enabled: true
  transforms:
    # ... 500+ lines of inline VRL ...
  sinks:
    kafka_out:
      type: kafka
      bootstrap_servers: "${KAFKA_BROKERS_SASL_SCRAM}"
      topic: enriched_syslog_land
      # ...

After (dfe-transform-vector — dfe-transform-vector-syslog.yaml):

replicaCount: 3

resources:
  requests:
    cpu: "2"
    memory: 2Gi
  limits:
    cpu: "4"
    memory: 4Gi

config:
  pipeline:
    name: syslog-enrichment
  source:
    topics:
      - raw_syslog_land
    group_id: dfe-transform-vector-syslog
  sink:
    topic: enriched_syslog_land
    key_field: .org_id
    compression: zstd

keda:
  maxReplicaCount: 20

Key differences:

• No inline VRL — transforms are separate YAML files the wrapper loads from config.transforms.dir (the image's /etc/dfe-transform-vector/transforms) or lists in config.transforms.files
• No persistence — Deployment (not StatefulSet), data dir loss just re-reads from Kafka
• SASL/TLS can sit in one values file every pipeline layers over, so each pipeline file carries only its own values
• Kafka brokers come from DFE_TRANSFORM_VECTOR_SOURCE_BROKERS and DFE_TRANSFORM_VECTOR_SINK_BROKERS. A ${VAR} in the config is not expanded: neither the wrapper nor Vector 0.57+ interpolates environment variables
• SASL credentials come from the Kafka Secret through DFE_TRANSFORM_VECTOR_{SOURCE,SINK}_SASL_{USERNAME,PASSWORD}, and the wrapper hands them to Vector's directory secret backend as owner-only files
• KEDA scales on CPU, with no Kafka lag trigger and no TriggerAuthentication

What Stays the Same

• Secrets: the chart creates none. The Kafka SASL secret comes from wherever your deployment already supplies it
• Node placement: set the chart's nodeSelector and tolerations to match your existing pool

What Changes

Element	Before	After
Chart source	helm.vector.dev v0.42.1	The thin chart each dfe-transform-vector release publishes, on scalo-service
Image	timberio/vector:0.48.0-debian	ghcr.io/hyperi-io/dfe-transform-vector
Workload type	StatefulSet	Deployment
Health endpoint	/health on :8686	/livez, /readyz on :9090, beside /metrics
Metrics	Vector native on :9090	Wrapper + merged Vector on :9090
Config format	Full Vector YAML (customConfig)	Big-dial config (source/sink/transforms)
Transform delivery	Inline VRL in customConfig	YAML files in transforms.dir or listed in transforms.files. The chart has no init container or download step: bake the files into your image or mount them at that path
DAG wiring	Manual (inline in values)	Auto-wired by wrapper
Hot-reload	Not supported	Poll-based file watcher + SIGHUP
Crash recovery	K8s pod restart only	Wrapper restarts Vector with backoff

Cutover Procedure

Step 1: Prepare Pipeline Values

For each existing Vector values file, write a dfe-transform-vector values file in the big-dial format above.

Step 2: Deploy in Parallel

Run both releases at once. The new pipeline uses a different consumer group ID, so it reads from the same source topic independently without affecting the existing pipeline.

Step 3: Validate

• Compare output topic records between old and new pipelines
• Monitor metrics: throughput, latency, error rates
• Check health endpoints respond correctly
• Verify KEDA scaling behaviour

Step 4: Switch Traffic

Once validated:

1. Scale down old pipeline replicas to 0
2. Update downstream consumers to read from new output topic (if topic name changed)
3. Monitor for 24-48 hours
4. Remove the old Vector release and its values file