DFE on Redpanda Cloud

DFE does NOT auto-deploy Redpanda Cloud -- you provision it, then point DFE at it with kafka.mode=external. Redpanda Cloud is the DFE provider redpanda-cloud, auth SASL_SSL with SCRAM-SHA-512 (the DFE credential contract; Redpanda Cloud supports both SCRAM-256 and 512, DFE mandates 512). Redpanda is Raft-native (no ZooKeeper, no KRaft sidecar) and ships a built-in schema registry.

Read the opinionated baseline in README.md first. Redpanda Cloud sits between MSK Provisioned and Confluent Cloud on how much it exposes: topic RF and partitions are settable (unlike Confluent/MSK), and the Dedicated / BYOC tiers accept cluster-level config; Serverless abstracts capacity away.

Note: this is Redpanda's MANAGED cloud, a different thing from the self-hosted Redpanda that DFE CAN auto-deploy in-cluster (kafka.provider=redpanda, BSL gate). See docs/deployment/kafka/redpanda-limitations.md for the self-hosted licensing story; Redpanda Cloud is a paid managed service, so the BSL "streaming service" clause is Redpanda's concern, not the deployer's.

Tiers

• Serverless -- fastest to stand up, elastic, capacity fully managed. Topic RF/partitions are managed; you set per-topic retention/compression/cleanup.
• Dedicated -- a managed cluster on Redpanda's cloud account. Topic RF and partitions are settable; cluster-level config (log_segment_size, log_retention_ms, default topic properties) is available.
• BYOC (Bring Your Own Cloud) -- the same, provisioned into YOUR cloud account/VPC. Same config surface as Dedicated.

What Redpanda Cloud lets you set vs manages

DFE default	Serverless	Dedicated / BYOC
Raft metadata (no ZooKeeper)	Managed	Managed
replication.factor=3	Managed (topic RF fixed by tier)	Settable per topic (replication_factor=3)
min.insync.replicas=2	Per-topic where supported; else Raft quorum	Per-topic where supported; else Raft quorum (majority ack)
unclean.leader.election off	N/A -- Raft never elects a non-quorum leader	N/A -- Raft
num.partitions=12	Per-topic (partition_count); auto-scales	Per-topic (partition_count=12)
compression.type=producer	Per-topic (compression.type); Redpanda defaults to producer	Per-topic; default producer
log.retention.hours=72	Per-topic retention.ms	Per-topic retention.ms, or cluster log_retention_ms
log.retention.bytes (unset)	Per-topic retention.bytes	Per-topic retention.bytes, or cluster retention_bytes
log.segment.bytes=512 MiB	Managed	Cluster log_segment_size (Redpanda property name)
cleanup.policy=delete	Per-topic (default delete)	Per-topic (default delete)
message.max.bytes (default)	Per-topic max.message.bytes	Per-topic max.message.bytes

Redpanda's Raft consensus makes unclean.leader.election moot (a leader always holds a committed quorum) and provides the RF3 durability equivalent via majority acknowledgement when the client sends acks=all (scalo default). Set min.insync.replicas per topic if you want a stricter explicit floor.

Terraform / OpenTofu -- Redpanda Cloud (SCRAM-SHA-512)

The redpanda provider authenticates with a Redpanda Cloud service-account client ID + secret (Organization IAM). Two shapes below: Serverless (fewest dials) and Dedicated/BYOC (full topic + cluster config). The SCRAM user is a redpanda_user with mechanism = "scram-sha-512" and its ACLs, against the cluster's Data Plane API URL (dataplane_api.url). There is no topic resource: dfe-engine creates every topic.

terraform {
  required_providers {
    redpanda = {
      source  = "redpanda-data/redpanda"
      version = "~> 2.3" # the line managed-kafka/redpanda-cloud pins
    }
  }
}

# Auth via env: REDPANDA_CLIENT_ID / REDPANDA_CLIENT_SECRET (service account).
provider "redpanda" {}

resource "redpanda_resource_group" "dfe" {
  name = "dfe"
}

# ---------- Option A: Serverless ----------
resource "redpanda_serverless_cluster" "dfe" {
  name              = "dfe-kafka-serverless"
  resource_group_id = redpanda_resource_group.dfe.id
  serverless_region = var.redpanda_serverless_region # e.g. "pro-us-east-1"
}

# The Data Plane API URL the user and ACL resources talk to.
locals {
  cluster_api_url = redpanda_serverless_cluster.dfe.dataplane_api.url
}

# ---------- Option B: Dedicated (managed on Redpanda's account) ----------
# resource "redpanda_network" "dfe" {
#   name              = "dfe"
#   resource_group_id = redpanda_resource_group.dfe.id
#   cloud_provider    = "aws"
#   region            = var.redpanda_region
#   cluster_type      = "dedicated"
#   cidr_block        = "10.20.0.0/20"
# }
# resource "redpanda_cluster" "dfe" {
#   name              = "dfe-kafka"
#   resource_group_id = redpanda_resource_group.dfe.id
#   network_id        = redpanda_network.dfe.id
#   cloud_provider    = "aws"
#   region            = var.redpanda_region
#   cluster_type      = "dedicated"
#   connection_type   = "public"
#   throughput_tier   = "tier-1-aws"
#   zones             = ["use1-az1", "use1-az2", "use1-az4"] # 3 AZs -> RF3
#   allow_deletion    = false
# }

# --- The DFE SCRAM-512 service account (SASL user) ---
resource "redpanda_user" "dfe_app" {
  name            = "dfe-kafka-user"
  password        = var.dfe_kafka_password
  mechanism       = "scram-sha-512" # DFE contract (provider default is scram-sha-256)
  cluster_api_url = local.cluster_api_url
}

# --- The user's grants, as managed-kafka/redpanda-cloud/acls.tf ---
# Topic * with every call dfe-engine makes, since it creates every topic at
# startup, and consumer groups fenced to the dfe- prefix. No topic resource.
resource "redpanda_acl" "dfe_topics" {
  for_each              = toset(["READ", "WRITE", "CREATE", "DESCRIBE", "DELETE", "ALTER", "DESCRIBE_CONFIGS", "ALTER_CONFIGS"])
  resource_type         = "TOPIC"
  resource_name         = "*"
  resource_pattern_type = "LITERAL"
  principal             = "User:${redpanda_user.dfe_app.name}"
  host                  = "*"
  operation             = each.value
  permission_type       = "ALLOW"
  cluster_api_url       = local.cluster_api_url
}

resource "redpanda_acl" "dfe_groups" {
  for_each              = toset(["READ", "DESCRIBE"])
  resource_type         = "GROUP"
  resource_name         = "dfe-"
  resource_pattern_type = "PREFIXED"
  principal             = "User:${redpanda_user.dfe_app.name}"
  host                  = "*"
  operation             = each.value
  permission_type       = "ALLOW"
  cluster_api_url       = local.cluster_api_url
}

output "cluster_api_url" {
  value = local.cluster_api_url
}

For the Dedicated / BYOC tier, point the local at redpanda_cluster.dfe.cluster_api_url, and set cluster-level defaults (log_segment_size = 536870912, log_retention_ms = 259200000, default_topic_partitions = 12, default_topic_replications = 3) via the cluster config surface -- these are the same Redpanda property names DFE's own self-hosted Redpanda CR uses (helm/charts/kafka/templates/redpanda.yaml).

Wiring back to DFE

• kafka.mode=external, kafka.external.bootstrap=<the SASL bootstrap host:port> (the seed brokers, not the HTTP cluster_api_url -- that URL is the admin/REST plane the Terraform provider uses; the client bootstrap is the Kafka SASL endpoint from the cluster's connectivity details).
• kafka.securityProtocol=SASL_SSL, the one dial every app chart and dfe-engine read their protocol from -- Redpanda Cloud serves SCRAM-SHA-512 over TLS only. On the aws root you set none of the mode, the endpoint or this by hand: DFE_KAFKA_PROVIDER=redpanda-cloud reaches bootstrap, which puts all three on the Argo cluster secret for the layer 2 appsets to hand on.
• Credentials: the redpanda_user name (username) + password go in the Vault-backed external secret; kafka.external.auth.type=scram.
• Client config: dfe kafka client-config --provider redpanda-cloud.

Verification -- what is and is not confirmed

Confirmed by 2026 research (sources in README): the redpanda provider is GA; redpanda_serverless_cluster (name + resource_group_id + serverless_region), redpanda_cluster (Dedicated/BYOC), redpanda_topic (partition_count, replication_factor, configuration map with cleanup.policy / retention.ms / etc.), and redpanda_user with mechanism = "scram-sha-512" are all current. Redpanda cluster property names (log_segment_size, log_retention_ms, default_topic_partitions, default_topic_replications) are confirmed. The DFE redpanda-cloud provider is SASL_SSL SCRAM-SHA-512 (scalo providers.rs).

NOT verified (no Redpanda Cloud account here, cannot deploy): the exact provider version to pin, the precise serverless_region / throughput_tier / zones strings for your account, whether Serverless honours per-topic replication_factor (it may manage RF and ignore the field), and the exact Kafka SASL bootstrap host to feed kafka.external.bootstrap (read it from the cluster's connectivity details after apply). Run tofu validate and plan against your account before apply.