Skip to main content

Redpanda as a DFE Kafka provider -- limitations and when to use it

DFE deploys two Kafka providers itself, in-cluster: Strimzi (default) and Redpanda (opt-in). Three more are managed clouds it does not deploy -- msk, confluent-cloud, redpanda-cloud -- covered in the neighbouring guides in this directory. This page is Redpanda-the-in-cluster-operator only. Redpanda is a Kafka-API-compatible broker with a much smaller footprint (no JVM, no ZooKeeper/KRaft sidecar, built-in schema registry). It is a strong choice for single/small/test and a lean option at scale -- but it carries constraints that must be a conscious decision. Redpanda is off unless kafka.redpanda.acceptLicense=true is set (the chart fails otherwise).

1. Licensing (the gating constraint)​

Redpanda Community Edition is licensed under the Business Source License (BSL) -- source-available, not open source. Sources: Redpanda licensing docs and the BSL announcement.

  • Core restriction: you may not offer Redpanda as a "Streaming or Queuing Service" to third parties (a commercial service where third parties cause topic creation). The BSL code converts to Apache-2.0 four years after each merge.
  • Internal / self-hosted production use is permitted.

What this means for DFE:

  • A customer self-hosting DFE with bundled Redpanda -- fine (internal use).
  • A HyperI-hosted, multi-tenant DFE SaaS that bundles Redpanda and lets tenants create topics -- likely breaches the BSL clause. Use Strimzi (Apache-2.0) for the hosted offering, or obtain a Redpanda Enterprise licence / legal sign-off.
  • Enterprise-only features (Tiered Storage, remote read replicas, continuous data balancing, Console SSO) require a paid licence (RCL). DFE uses none of these -- it does its own S3 archival via dfe-archiver.

2. Kafka-API compatibility caveats (vs DFE's usage)​

Redpanda implements the Kafka wire protocol (clients >= 0.11), so DFE's produce/consume/consumer-group usage works without code changes. Known gaps:

CaveatRelevant to DFE?
Kafka Streams unsupportedNo -- DFE uses its own Rust transforms, not Kafka Streams
Transactions: server-side KIP-890 absent (4.x clients fall back to old protocol)Only if DFE relies on exactly-once Kafka transactions; confirm in the loader (DFE is at-least-once)
One SASL/SCRAM mechanism per user (no simultaneous 256+512)No -- DFE uses SCRAM-SHA-512 only
HTTP Proxy cannot do topic/ACL admin CRUDNo -- DFE creates topics via the admin API, not pandaproxy
No live KRaft<->Redpanda metadata migrationNo -- greenfield
May lag newest Kafka KIPs by monthsLow risk for basic produce/consume

Upside: built-in Schema Registry + HTTP proxy per node, no JVM/ZK -- smaller and simpler than Strimzi for single/small deployments.

3. Deployment status in DFE charts​

Redpanda is deployed via the Redpanda operator (Redpanda + User CRDs, cluster.redpanda.com/v1alpha2) -- the same operator+thin-CR model DFE uses for Strimzi. The operator is installed only on an explicit opt-in: the dfe-redpanda-operator ApplicationSet keys on the cluster label dfe.hyperi.io/kafka-provider: redpanda (BSL stays off any cluster that has not chosen it). Pinned in versions.yaml (redpanda-operator 26.2.3, broker redpanda-version v26.2.2).

  • Auth: SASL/SCRAM-SHA-512, on from first boot (the DFE kafka standard), TLS off on the internal listener -- matching Strimzi's plain SCRAM listener -- with the Kafka API on 9093. The operator generates and manages the bootstrap superuser, and the dfe-kafka-user User CR reads its password from the store-backed dfe-kafka-user-store Secret the Strimzi KafkaUser reads. That Secret carries password alone, so the chart writes username, password and sasl.mechanism into dfe-kafka-user in the app namespaces, as it does for Strimzi. With user.password.fromSecretsStore=false the operator generates the password into a broker-namespace Secret named after the user, and the chart merges the other two keys into it.
  • single (small/test): one-broker Redpanda CR. Live-validated end to end on the on-prem reference cluster (SCRAM produce->consume via the matrix harness).
  • cluster: multi-broker Redpanda CR (statefulset.replicas); the operator owns Raft/seed bootstrap (no hand-rolled seed_servers). Validate at the target scale before relying on it. The secrets config must set kafka_provider=redpanda, or the store path kafka/redpanda is never seeded and dfe-kafka-user-store never syncs. bootstrapServers in profile-scale.yaml names Strimzi's Service, so override it with the Redpanda cluster's in your own values.
  • Known operator bug (workaround in place): the User controller's clusterRef SRV discovery fails when SASL is enabled (redpanda-operator #1130, "Operator fails to load credentials from clusterRef when SASL is enabled"). The User CR therefore uses spec.cluster.staticConfiguration (explicit kafka + admin endpoints, authenticating as the bootstrap superuser) instead of clusterRef. Revert to clusterRef once the upstream bug is fixed.
  • MSK / external: not applicable to Redpanda -- use kafka.mode=external with a Redpanda Cloud endpoint, or MSK with the Strimzi/external path.

4. Decision summary​

  • Default to Strimzi (Apache-2.0, CNCF, mature, fully cluster-validated here).
  • Choose Redpanda for footprint-sensitive self-hosted single/small/test, with acceptLicense=true, having read the BSL constraint above.
  • Do not bundle Redpanda into a HyperI-hosted multi-tenant SaaS without legal sign-off.