# Migration Guide: Vector → dfe-transform-vector

> This guide covers migrating existing Vector pipelines from the official
> `timberio/vector` Helm chart to the `dfe-transform-vector` wrapper.

## Pipeline Values Migration

A values file for the official Vector chart carries the whole Vector config.
dfe-transform-vector's chart takes the "big-dial" config pattern instead, one
values file per pipeline. Both releases can run side by side while you move.

### Values Format

**Before** (official Vector chart — `vector-syslog.yaml`):

```yaml
replicas: 3
image:
  tag: "0.48.0-debian"
persistence:
  size: 20Gi
resources:
  requests:
    cpu: "2"
    memory: 2Gi
  limits:
    cpu: "4"
    memory: 4Gi
customConfig:
  sources:
    kafka_in:
      type: kafka
      bootstrap_servers: "${KAFKA_BROKERS_SASL_SCRAM}"
      group_id: dfe-vector-syslog
      topics: [raw_syslog_land]
      sasl:
        enabled: true
        mechanism: SCRAM-SHA-512
        username: "${KAFKA_SASL_USERNAME}"
        password: "${KAFKA_SASL_PASSWORD}"
      tls:
        enabled: true
  transforms:
    # ... 500+ lines of inline VRL ...
  sinks:
    kafka_out:
      type: kafka
      bootstrap_servers: "${KAFKA_BROKERS_SASL_SCRAM}"
      topic: enriched_syslog_land
      # ...
```

**After** (dfe-transform-vector — `dfe-transform-vector-syslog.yaml`):

```yaml
replicaCount: 3

resources:
  requests:
    cpu: "2"
    memory: 2Gi
  limits:
    cpu: "4"
    memory: 4Gi

config:
  pipeline:
    name: syslog-enrichment
  source:
    topics:
      - raw_syslog_land
    group_id: dfe-transform-vector-syslog
  sink:
    topic: enriched_syslog_land
    key_field: .org_id
    compression: zstd

keda:
  maxReplicaCount: 20
```

Key differences:

- **No inline VRL** — transforms are separate YAML files the wrapper loads from `config.transforms.dir` (the image's `/etc/dfe-transform-vector/transforms`) or lists in `config.transforms.files`
- **No persistence** — Deployment (not StatefulSet), data dir loss just re-reads from Kafka
- **SASL/TLS** can sit in one values file every pipeline layers over, so each pipeline file carries only its own values
- **Kafka brokers** come from `DFE_TRANSFORM_VECTOR_SOURCE_BROKERS` and
  `DFE_TRANSFORM_VECTOR_SINK_BROKERS`. A `${VAR}` in the config is not expanded:
  neither the wrapper nor Vector 0.57+ interpolates environment variables
- **SASL credentials** come from the Kafka Secret through
  `DFE_TRANSFORM_VECTOR_{SOURCE,SINK}_SASL_{USERNAME,PASSWORD}`, and the wrapper hands
  them to Vector's directory secret backend as owner-only files
- **KEDA** scales on CPU, with no Kafka lag trigger and no
  TriggerAuthentication

### What Stays the Same

- **Secrets**: the chart creates none. The Kafka SASL secret comes from wherever your deployment already supplies it
- **Node placement**: set the chart's `nodeSelector` and `tolerations` to match your existing pool

### What Changes

| Element | Before | After |
|---------|--------|-------|
| Chart source | `helm.vector.dev` v0.42.1 | The thin chart each `dfe-transform-vector` release publishes, on scalo-service |
| Image | `timberio/vector:0.48.0-debian` | `ghcr.io/hyperi-io/dfe-transform-vector` |
| Workload type | StatefulSet | Deployment |
| Health endpoint | `/health` on :8686 | `/livez`, `/readyz` on :9090, beside `/metrics` |
| Metrics | Vector native on :9090 | Wrapper + merged Vector on :9090 |
| Config format | Full Vector YAML (customConfig) | Big-dial config (source/sink/transforms) |
| Transform delivery | Inline VRL in `customConfig` | YAML files in `transforms.dir` or listed in `transforms.files`. The chart has no init container or download step: bake the files into your image or mount them at that path |
| DAG wiring | Manual (inline in values) | Auto-wired by wrapper |
| Hot-reload | Not supported | Poll-based file watcher + SIGHUP |
| Crash recovery | K8s pod restart only | Wrapper restarts Vector with backoff |

## Cutover Procedure

### Step 1: Prepare Pipeline Values

For each existing Vector values file, write a dfe-transform-vector values file
in the big-dial format above.

### Step 2: Deploy in Parallel

Run both releases at once. The new pipeline uses a different consumer group
ID, so it reads from the same source topic independently without affecting
the existing pipeline.

### Step 3: Validate

- Compare output topic records between old and new pipelines
- Monitor metrics: throughput, latency, error rates
- Check health endpoints respond correctly
- Verify KEDA scaling behaviour

### Step 4: Switch Traffic

Once validated:

1. Scale down old pipeline replicas to 0
2. Update downstream consumers to read from new output topic (if topic name changed)
3. Monitor for 24-48 hours
4. Remove the old Vector release and its values file