DFE on Redpanda Cloud

DFE doesn't deploy Redpanda Cloud. You provision it, then point DFE at it with kafka.mode=external. Redpanda Cloud is the DFE provider redpanda-cloud, with SASL_SSL and SCRAM-SHA-512 auth. Redpanda Cloud supports both SCRAM-256 and 512, and DFE mandates 512. Redpanda is Raft-native (no ZooKeeper, no KRaft sidecar) and ships a built-in schema registry.

Read the baseline in README.md first. Dedicated and BYOC let you set topic RF and partitions and accept cluster-level config (see Tiers). Serverless manages capacity.

This is Redpanda's MANAGED cloud, which isn't the self-hosted Redpanda DFE can deploy in-cluster (kafka.provider=redpanda, BSL gate). docs/deployment/kafka/redpanda-limitations.md has the self-hosted licensing story. Redpanda Cloud is a paid managed service, so the BSL "streaming service" clause is Redpanda's concern and not the deployer's.

Tiers

Serverless is elastic and its capacity is fully managed. Topic RF and partitions are managed, and you set per-topic retention, compression and cleanup.
Dedicated is a managed cluster on Redpanda's cloud account. Topic RF and partitions are settable, and cluster-level config (log_segment_size, log_retention_ms, default topic properties) is available.
BYOC (Bring Your Own Cloud) is the same, provisioned into YOUR cloud account and VPC. It has the same config surface as Dedicated.

What Redpanda Cloud lets you set vs manages

DFE default	Serverless	Dedicated / BYOC
Raft metadata (no ZooKeeper)	Managed	Managed
`replication.factor=3`	Managed (topic RF fixed by tier)	Settable per topic (`replication_factor=3`)
`min.insync.replicas=2`	Per-topic where supported, else Raft quorum	Per-topic where supported, else Raft quorum (majority ack)
`unclean.leader.election` off	N/A. Raft never elects a non-quorum leader	N/A (Raft)
`num.partitions=12`	Per-topic (`partition_count`), auto-scales	Per-topic (`partition_count=12`)
`compression.type=producer`	Per-topic (`compression.type`), and Redpanda defaults to `producer`	Per-topic, default `producer`
`log.retention.hours=72`, what `dfe-ops init` writes to the dial for a managed broker	Per-topic `retention.ms`	Per-topic `retention.ms`, or cluster `log_retention_ms`
`log.retention.bytes` (unset)	Per-topic `retention.bytes`	Per-topic `retention.bytes`, or cluster `retention_bytes`
`log.segment.bytes` (left at the default)	Managed	Cluster `log_segment_size` (Redpanda property name), left at the default
`cleanup.policy=delete`	Per-topic (default `delete`)	Per-topic (default `delete`)
`message.max.bytes` (default)	Per-topic `max.message.bytes`	Per-topic `max.message.bytes`

Redpanda's Raft consensus makes unclean.leader.election moot, because a leader always holds a committed quorum. It gives you the RF3 durability equivalent through majority acknowledgement when the client sends acks=all (the scalo default). Set min.insync.replicas per topic if you want a stricter explicit floor.

OpenTofu: Redpanda Cloud (SCRAM-SHA-512)

The redpanda provider authenticates with a Redpanda Cloud service-account client ID and secret (Organization IAM). There are two shapes below, Serverless (fewest dials) and Dedicated/BYOC (full topic and cluster config). The SCRAM user is a redpanda_user with mechanism = "scram-sha-512" and its ACLs, against the cluster's Data Plane API URL (dataplane_api.url). There's no topic resource, because dfe-engine creates every topic.

terraform {
  required_providers {
    redpanda = {
      source  = "redpanda-data/redpanda"
      version = "~> 2.3" # the line managed-kafka/redpanda-cloud pins
    }
  }
}

# Auth via env: REDPANDA_CLIENT_ID / REDPANDA_CLIENT_SECRET (service account).
provider "redpanda" {}

resource "redpanda_resource_group" "dfe" {
  name = "dfe"
}

# ---------- Option A: Serverless ----------
resource "redpanda_serverless_cluster" "dfe" {
  name              = "dfe-kafka-serverless"
  resource_group_id = redpanda_resource_group.dfe.id
  serverless_region = var.redpanda_serverless_region # e.g. "pro-us-east-1"
}

# The Data Plane API URL the user and ACL resources talk to.
locals {
  cluster_api_url = redpanda_serverless_cluster.dfe.dataplane_api.url
}

# ---------- Option B: Dedicated (managed on Redpanda's account) ----------
# resource "redpanda_network" "dfe" {
#   name              = "dfe"
#   resource_group_id = redpanda_resource_group.dfe.id
#   cloud_provider    = "aws"
#   region            = var.redpanda_region
#   cluster_type      = "dedicated"
#   cidr_block        = "10.20.0.0/20"
# }
# resource "redpanda_cluster" "dfe" {
#   name              = "dfe-kafka"
#   resource_group_id = redpanda_resource_group.dfe.id
#   network_id        = redpanda_network.dfe.id
#   cloud_provider    = "aws"
#   region            = var.redpanda_region
#   cluster_type      = "dedicated"
#   connection_type   = "public"
#   throughput_tier   = "tier-1-aws"
#   zones             = ["use1-az1", "use1-az2", "use1-az4"] # 3 AZs -> RF3
#   allow_deletion    = false
# }

# --- The DFE SCRAM-512 service account (SASL user) ---
resource "redpanda_user" "dfe_app" {
  name            = "dfe-kafka-user"
  password        = var.dfe_kafka_password
  mechanism       = "scram-sha-512" # DFE contract (provider default is scram-sha-256)
  cluster_api_url = local.cluster_api_url
}

# --- The user's grants, as managed-kafka/redpanda-cloud/acls.tf ---
# Topic * with every call dfe-engine makes, since it creates every topic at
# startup, and consumer groups fenced to the dfe- prefix. No topic resource.
resource "redpanda_acl" "dfe_topics" {
  for_each              = toset(["READ", "WRITE", "CREATE", "DESCRIBE", "DELETE", "ALTER", "DESCRIBE_CONFIGS", "ALTER_CONFIGS"])
  resource_type         = "TOPIC"
  resource_name         = "*"
  resource_pattern_type = "LITERAL"
  principal             = "User:${redpanda_user.dfe_app.name}"
  host                  = "*"
  operation             = each.value
  permission_type       = "ALLOW"
  cluster_api_url       = local.cluster_api_url
}

resource "redpanda_acl" "dfe_groups" {
  for_each              = toset(["READ", "DESCRIBE"])
  resource_type         = "GROUP"
  resource_name         = "dfe-"
  resource_pattern_type = "PREFIXED"
  principal             = "User:${redpanda_user.dfe_app.name}"
  host                  = "*"
  operation             = each.value
  permission_type       = "ALLOW"
  cluster_api_url       = local.cluster_api_url
}

output "cluster_api_url" {
  value = local.cluster_api_url
}

For the Dedicated / BYOC tier, point the local at redpanda_cluster.dfe.cluster_api_url. Then set cluster-level defaults (log_retention_ms = 259200000, default_topic_partitions = 12, default_topic_replications = 3) through the cluster config surface. They're the same Redpanda property names DFE's own self-hosted Redpanda CR uses (helm/charts/kafka/templates/redpanda.yaml).

Wiring back to DFE

kafka.mode=external, kafka.external.bootstrap=<the SASL bootstrap host:port>. That's the seed brokers. It isn't the HTTP cluster_api_url, which is the admin/REST plane the Terraform provider uses. The client bootstrap is the Kafka SASL endpoint from the cluster's connectivity details.
kafka.securityProtocol=SASL_SSL, the one dial every app chart and dfe-engine read their protocol from. Redpanda Cloud serves SCRAM-SHA-512 over TLS only. On the aws root you set none of the mode, the endpoint or this by hand. DFE_KAFKA_PROVIDER=redpanda-cloud reaches bootstrap, which puts all three on the Argo cluster secret for the layer 2 appsets to hand on.
Credentials: the redpanda_user name (username) and password go in the external secret in the deployment's secrets store (Secrets Manager on AWS, OpenBao elsewhere), with kafka.external.auth.type=scram.
Client config: dfe kafka client-config --provider redpanda-cloud.

Verification: what is and isn't confirmed

Confirmed by 2026 research (sources in the README):

The redpanda provider is GA.
redpanda_serverless_cluster (name, resource_group_id, serverless_region), redpanda_cluster (Dedicated/BYOC), redpanda_topic (partition_count, replication_factor, a configuration map with cleanup.policy, retention.ms and so on) and redpanda_user with mechanism = "scram-sha-512" are all current.
The Redpanda cluster property names log_segment_size, log_retention_ms, default_topic_partitions and default_topic_replications are confirmed.
The DFE redpanda-cloud provider is SASL_SSL SCRAM-SHA-512 (scalo providers.rs).

NOT verified for the hand-written snippet above, which the managed-kafka/redpanda-cloud module path does not exercise:

The precise serverless_region, throughput_tier and zones strings for your account.
Whether Serverless honours per-topic replication_factor. It may manage RF and ignore the field.
The exact Kafka SASL bootstrap host to feed kafka.external.bootstrap. Read it from the cluster's connectivity details after apply.

Run tofu validate and plan against your account before you apply.

Tiers​

What Redpanda Cloud lets you set vs manages​

OpenTofu: Redpanda Cloud (SCRAM-SHA-512)​

Wiring back to DFE​

Verification: what is and isn't confirmed​

Tiers

What Redpanda Cloud lets you set vs manages

OpenTofu: Redpanda Cloud (SCRAM-SHA-512)

Wiring back to DFE

Verification: what is and isn't confirmed