Edge exposure and authentication -- cloud specifics

On a cloud deploy the Gateway Service is internet-facing, and public web access is default-deny. The model, the identity plane and on-prem behaviour are in edge-auth.md.

Internet-facing defaults

envoyGateway.service.internetFacing is true in edge-aws.yaml, edge-gcp.yaml and edge-azure.yaml, because each cloud makes a LoadBalancer public unless told otherwise.

Public web access is default-deny, per exposure group. A route's class puts it in a group. dfe-ui, dfe-hyperdx and the engine's browser families are console. The engine's CLI families, /openapi.json and SCIM are api. The admin UIs are admin. Every route of a group, on its internal name and its public one, is refused at Envoy unless the client address is on exposure.groups.<group>.allow. The group's deny list is refused first.

exposure.groups.<group>.allow	The group's routes	Said by
empty	403 to every address outside exposure.selfRanges	NOTES, dfe-ops access-summary
ranges	the ranges, and exposure.selfRanges	NOTES, dfe-ops access-summary
a /0 of either family	every address	a WARNING in both

Every web route the chart renders carries its own SecurityPolicy with its group's rule, and the OIDC ones AND it with their group check. The Gateway-wide policy, dfe-edge-fence, refuses every address, so it holds only a route another namespace attaches. Ingest routes (otel, and receiver on exposure.groups.ingest.door: gateway) carry the ingest group's rule the same way. The :80 redirect carries an explicit allow, since it serves nothing but a 301 to the fenced https side.

One Service carries every listener, so the load balancer can't tell the groups apart. Its source ranges are the union of the enabled web groups' allow lists, the ingest group's while an ingest route renders, exposure.selfRanges and envoyGateway.service.loadBalancerSourceRanges. A group that admits every address leaves it open. The Envoy Service carries the web groups' state as dfe.hyperi.io/edge-fence: console=<state>,api=<state>,admin=<state>, and the CIDRs each group's routes admit, with exposure.selfRanges, as JSON in dfe.hyperi.io/edge-fence-ranges.

exposure.selfRanges is the cluster's own NAT egress, which the aws root writes onto the cluster secret. The deployer's lists live in the deploy repo's infra/common.yaml, where the engine can change them at runtime.

exposure.groups.admin.enabled defaults off on the same three flavours, and argocd/values/aws.yaml sets oidc.enabled: true. Argo CD, Kafbat, Forgejo, the links page and Cruise Control all stay off the public load balancer until an operator's overlay turns the group on. dfe-hyperdx is class product, beside the console that frames it, so the switch never reaches it. The chart's render guard (envoy-gateway-config.validateUi) refuses the admin group on an internet-facing Service with an empty exposure.groups.admin.allow, unless allowPublic: true. It also refuses allowPublic without an edge OIDC provider (oidc.enabled and an oidc.providers entry).

Upgrading a cloud deployment

• Set exposure.groups.console.allow (and api.allow, admin.allow as needed) in the deploy repo's infra/common.yaml before upgrading. An internet-facing Gateway with an empty list answers 403 on that group's routes from the first sync.
• ui.allowed_cidrs, exposure.infraUisExternal and ui.public.<admin UI> are retired and refuse to render, each naming its replacement.
• The trusted ranges a deployment already runs with live on its Argo CD cluster secret, in dfe.hyperi.io/edge_trusted_proxy_cidrs, and stay trusted until bootstrap.sh runs again. Only bootstrap writes that annotation, and dfe-ops refresh and dfe-ops upgrade leave it alone. Re-run bootstrap after upgrading, or remove the annotation, so a load balancer's subnets stop being believed.

The client address the fence judges

The fence judges the TCP peer. No listener believes X-Forwarded-For unless ui.trusted_proxy_cidrs names the hops in front. So neither a client nor a host beside the load balancer, such as the tunnel forwarder in an AWS public subnet, can claim a listed address. With no client-IP detection, Envoy Gateway v1.9.2 sets use_remote_address and trusts no hop (internal/xds/translator/listener.go).

Flavour	Load balancer	Envoy's TCP peer
AWS	NLB with IP targets and preserve_client_ip.enabled=true (edge-aws.yaml)	the client
GCP	GKE's passthrough Network Load Balancer, externalTrafficPolicy: Local (Envoy Gateway's default)	the client
Azure	Azure Load Balancer, externalTrafficPolicy: Local	the client
On-prem	MetalLB or another L4 pool, externalTrafficPolicy: Local	the client, or a node address where the pool SNATs

A CDN in front of the public hostnames writes the client into X-Forwarded-For and connects from its own ranges. Name those ranges in ui.trusted_proxy_cidrs (the dial's edge.product.trusted_proxy_cidrs). The public listeners then read the header right to left when the peer is inside the list and take the first address outside it, while every other connection is judged by its own address. The internal listener never reads the header. List only a proxy that writes the header itself. Never an L4 load balancer, and never a subnet other hosts share.

Public hostnames on a cloud deploy

Everything DFE uses internally is private. The UIs are the exception, because their users sit outside the VPC, and the gateway chart's ui: values expose them. Nothing in it acts until ui.public_domain (the dial's edge.product.domain, itself the deployment's dns.public_zone) names a delegated public zone.

The keys below are the CHART's. The dial states the same decisions under edge.product and edge.admin_uis (deployment.example.yaml), where rate_limit, waf and tls are these values verbatim and the rest is the dial's own spelling. ui.rate_limit.per_client_requests is the chart's alone.

Key	Default	What it decides
ui.public_domain	""	The public zone the UIs answer on. Empty renders no public half.
ui.public.<name>	dfe_ui: true, hyperdx: false	Which product UIs get a public hostname.
exposure.groups.admin.tools.<route>	every tool false	Which admin UIs get a public hostname, once the admin group is on.
exposure.groups.<group>.allow, .deny	[] (nobody, internet-facing)	Who may reach the group's routes, at Envoy AND, as a union, at the load balancer.
ui.trusted_proxy_cidrs	"" (no forwarded header believed)	A CDN or other L7 proxy in front of the public hostnames, whose X-Forwarded-For names the client.
ui.rate_limit	on, 6000/Minute, local	Each public route's shared bucket, per proxy replica. A distributed credential-stuffing run hits it before the login.
ui.rate_limit.per_client_requests	1200 (per unit)	A bucket per client address, per route, per proxy replica, on every public route and on every internal-name web route of an internet-facing Service, so one caller's dashboard cannot empty dfe-hyperdx for the rest.
envoyGateway.connectionLimit	on, 10000	Concurrent connections each listener takes per proxy replica.
ui.waf.mode	none	Only none renders.
ui.tls	"1.2", HSTS on	The public TLS floor and Strict-Transport-Security.

A public UI gets its OWN listener, its own publicly trusted certificate and its own HTTPRoute <route>-public. It isn't a second hostname on the internal route, because the policies that make a public route safe must not reach the internal one. Internal routes keep the internal CA. The public hostnames hold a Let's Encrypt certificate issued by DNS-01 on the public zone through cert-manager's Pod Identity role (tls.public).

What fails the render

The render fails, rather than shipping a weaker edge, for:

• A public UI with no authentication. Edge OIDC (edgePolicy, or oidc.targetRoutes naming the route, with a provider configured) counts, as do the three app-side schemes above: dfe-ui's NextAuth, Kafbat's own OIDC, dfe-hyperdx's dfe_token cookie. Nothing else does.
• The receiver route on an internet-facing Service with no caller auth. The chart can't read the receiver's config.server.auth.mode, so routes.receiver.authMode restates it, and only header or bearer renders. Envoy terminates TLS, so mtls never sees a client certificate.
• internetFacing: false on a load balancer the cloud makes public. The key is declared, not derived, so false is refused beside service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing. On GCP or Azure (cloud or global.cloud) it's refused on a LoadBalancer Service with no networking.gke.io/load-balancer-type: Internal or service.beta.kubernetes.io/azure-load-balancer-internal: "true".
• An entry in any group list, selfRanges or trusted_proxy_cidrs that isn't a CIDR range, such as a bare address. Envoy Gateway and the load balancer would refuse it at admission, failing the whole gateway sync.
• The admin group on an internet-facing Service with no allow list, unless allowPublic is true with an edge OIDC provider.
• A retired key: ui.allowed_cidrs, exposure.infraUisExternal or ui.public.<admin UI>, each refused with its replacement named.
• A /0 of either family in trusted_proxy_cidrs. Every peer is then trusted, so Envoy takes the client from the leftmost X-Forwarded-For entry, which the caller writes. render_dial.py, bootstrap.sh and the aws root refuse it too.
• rate_limit.scope: global, which needs a Redis backend named in the EnvoyGateway install config.
• rate_limit.requests or rate_limit.per_client_requests that isn't an unquoted whole number from 1 to 4294967295, the range Envoy Gateway admits. 0 would refuse every caller, and the old dial's quoted "300" is a string.
• rate_limit.requests at or below rate_limit.per_client_requests. Envoy spends the lower-rate bucket first, so one caller could empty the shared one for every public user.
• An engine route whose internal path list is empty. A rule with no matches answers every path on the dfe hostname.
• waf.mode other than none. A WAF terminates TLS above Envoy, which moves the public certificate to the cloud's certificate store.
• A ui.public.<name> with no route, rather than the flag doing nothing.

On AWS the load-balancer half of the filter is loadBalancerSourceRanges, which the AWS Load Balancer Controller turns into the NLB's frontend security group. That group can only be attached at creation, so the controller must own the Service from the start. See the annotations and their reasons in argocd/values/edge-aws.yaml. The ranges fence the WHOLE front door, ingest included, because one Service carries every listener.

The engine answers on two routes on its one hostname. dfe-engine carries the browser families (routes.dfeEngine.browserFamilies) for the console group. dfe-engine-cli carries the CLI families, /openapi.json, the Swagger pages where the engine serves them and, with exposure.groups.api.scim, SCIM, for the api group. The edge OIDC login fronts dfe-engine alone, because the CLI can't follow an interactive redirect. The engine authenticates every call itself.