# Migration Guide: Vector → dfe-transform-vector

> This guide covers migrating existing Vector pipelines from the official
> `timberio/vector` Helm chart to the `dfe-transform-vector` wrapper.

## Pipeline Values Migration

Existing pipeline values files in the `pipelines` repo follow the official
Vector chart format. The new format uses the "big-dial" config pattern.

### File Naming

Old: `pipelines/<namespace>/vector-<pipeline>.yaml`
New: `pipelines/<namespace>/dfe-transform-vector-<pipeline>.yaml`

The ArgoCD ApplicationSet scans for `dfe-transform-vector-*.yaml` files.
Both old and new pipelines can coexist during migration.

### Values Format

**Before** (official Vector chart — `vector-syslog.yaml`):

```yaml
replicas: 3
image:
  tag: "0.48.0-debian"
persistence:
  size: 20Gi
resources:
  requests:
    cpu: "2"
    memory: 2Gi
  limits:
    cpu: "4"
    memory: 4Gi
customConfig:
  sources:
    kafka_in:
      type: kafka
      bootstrap_servers: "${KAFKA_BROKERS_SASL_SCRAM}"
      group_id: dfe-vector-syslog
      topics: [raw_syslog_land]
      sasl:
        enabled: true
        mechanism: SCRAM-SHA-512
        username: "${KAFKA_SASL_USERNAME}"
        password: "${KAFKA_SASL_PASSWORD}"
      tls:
        enabled: true
  transforms:
    # ... 500+ lines of inline VRL ...
  sinks:
    kafka_out:
      type: kafka
      bootstrap_servers: "${KAFKA_BROKERS_SASL_SCRAM}"
      topic: enriched_syslog_land
      # ...
```

**After** (dfe-transform-vector — `dfe-transform-vector-syslog.yaml`):

```yaml
replicaCount: 3

resources:
  requests:
    cpu: "2"
    memory: 2Gi
  limits:
    cpu: "4"
    memory: 4Gi

config:
  pipeline:
    name: syslog-enrichment
  source:
    topics:
      - raw_syslog_land
    group_id: dfe-transform-vector-syslog
  sink:
    topic: enriched_syslog_land
    key_field: .org_id
    compression: zstd

keda:
  maxReplicaCount: 20
```

Key differences:

- **No inline VRL** — transforms are separate YAML files the wrapper loads from `config.transforms.dir` (the image's `/etc/dfe-transform-vector/transforms`) or lists in `config.transforms.files`
- **No persistence** — Deployment (not StatefulSet), data dir loss just re-reads from Kafka
- **SASL/TLS** inherited from `common.yaml` — only override pipeline-specific values
- **Kafka brokers** come from `DFE_TRANSFORM_SOURCE_BROKERS` and
  `DFE_TRANSFORM_SINK_BROKERS`. A `${VAR}` in the config is not expanded:
  neither the wrapper nor Vector 0.57+ interpolates environment variables
- **SASL credentials** come from the Kafka Secret through
  `DFE_TRANSFORM_{SOURCE,SINK}_SASL_{USERNAME,PASSWORD}`, and the wrapper hands
  them to Vector's directory secret backend as owner-only files
- **KEDA** scales on CPU, with no Kafka lag trigger and no
  TriggerAuthentication

### What Stays the Same

- **Secrets**: the chart creates none. The Kafka SASL secret comes from wherever your deployment already supplies it
- **Node placement**: set the chart's `nodeSelector` and `tolerations` to match your existing pool

### What Changes

| Element | Before | After |
|---------|--------|-------|
| Chart source | `helm.vector.dev` v0.42.1 | The thin chart each `dfe-transform-vector` release publishes, on scalo-service |
| Image | `timberio/vector:0.48.0-debian` | `ghcr.io/hyperi-io/dfe-transform-vector` |
| Workload type | StatefulSet | Deployment |
| Health endpoint | `/health` on :8686 | `/livez`, `/readyz` on :9090, beside `/metrics` |
| Metrics | Vector native on :9090 | Wrapper + merged Vector on :9090 |
| Config format | Full Vector YAML (customConfig) | Big-dial config (source/sink/transforms) |
| Transform delivery | Inline VRL in `customConfig` | YAML files in `transforms.dir` or listed in `transforms.files`. The chart has no init container or download step: bake the files into your image or mount them at that path |
| DAG wiring | Manual (inline in values) | Auto-wired by wrapper |
| Hot-reload | Not supported | Poll-based file watcher + SIGHUP |
| Crash recovery | K8s pod restart only | Wrapper restarts Vector with backoff |

## Cutover Procedure

See the cutover plan section below for step-by-step parallel deployment
and traffic switching procedure.

### Step 1: Prepare Pipeline Values

For each existing `vector-<name>.yaml`, create a corresponding
`dfe-transform-vector-<name>.yaml` with big-dial config format.

### Step 2: Deploy in Parallel

Both ApplicationSets run simultaneously. The new pipeline uses a
different consumer group ID, so it reads from the same source topic
independently without affecting the existing pipeline.

### Step 3: Validate

- Compare output topic records between old and new pipelines
- Monitor metrics: throughput, latency, error rates
- Check health endpoints respond correctly
- Verify KEDA scaling behaviour

### Step 4: Switch Traffic

Once validated:

1. Scale down old pipeline replicas to 0
2. Update downstream consumers to read from new output topic (if topic name changed)
3. Monitor for 24-48 hours
4. Remove old pipeline values file from pipelines repo

### Step 5: Cleanup

After all pipelines migrated:

1. Remove old `vector.yaml` ApplicationSet from dfe-core
2. Remove old `gitOps/addons/helm/vector/` directory
3. Remove `vector_project.yaml` ArgoCD project (or update to cover new apps)