Cloud and SaaS Setup Guides

Each guide here covers what a provider administrator configures so dfe-fetcher can pull data from that provider: the access it needs, how to grant it least-privilege, where to put credentials, and how to verify the connection. Every source is read-side: M365 starts its activity-feed subscription and Pub/Sub acknowledges what it delivered. Nothing else changes provider state.

For the fetcher's own configuration (scheduler, output, credential resolution, the sources.db and sources.file blocks), see config.example.yaml. The architecture and the profile grammar are in ../internals.md.

Credentials in Kubernetes

The released chart mounts one Secret, Kafka's: its username, password and sasl.mechanism keys become DFE_FETCHER_KAFKA_SASL_USER, DFE_FETCHER_KAFKA_SASL_PASSWORD and DFE_FETCHER_KAFKA_SASL_MECHANISM. Name an existing Secret with the chart value secrets.kafka.existingSecret.

A provider's credentials are yours to supply, through the chart's extraEnv (a map of variable name to value or valueFrom) or extraEnvFrom (a list of envFrom sources) values. Two shapes work, and each guide's Environment Variables section names the fields:

• set the field itself, DFE_FETCHER_SOURCES__<BLOCK>__<FIELD>, from a Secret key, e.g. DFE_FETCHER_SOURCES__AWS__SECRET_ACCESS_KEY
• write the field as an env:NAME spec in the config, and load a Secret whose keys are those names with extraEnvFrom

extraEnv:
  DFE_FETCHER_SOURCES__AZURE__CLIENT_SECRET:
    valueFrom:
      secretKeyRef:
        name: fetcher-azure
        key: client-secret
extraEnvFrom:
  - secretRef:
      name: fetcher-credentials

An empty value counts as unset, so a Secret key left blank leaves the field unset rather than signing requests with an empty key.

Providers

Provider	Data it pulls	Guide
AWS	Security and operational logs	aws.md
Azure	Platform and security logs	azure.md
Microsoft 365	Audit and security logs	m365.md
Google Cloud	Audit and security logs	gcp.md
Google Cloud Pub/Sub	Subscription streaming	gcp-pubsub.md
Google Workspace	Admin and audit logs	google-workspace.md
Okta	System log	okta.md
Cisco Duo	Admin and authentication logs	duo.md
CrowdStrike	Falcon detections and events	crowdstrike.md
Cloudflare	Audit logs	cloudflare.md
Datadog	Audit trail and security signals	datadog.md
Bitwarden	Event logs	bitwarden.md
1Password	Audit and sign-in events	onepassword.md
Slack	Audit logs	slack.md
GitHub	Audit log	github.md
Salesforce	Event monitoring logs	salesforce.md
runZero	Asset inventory snapshots	runzero.md
Object store	Buckets (S3 and compatible)	object-store.md
Package registries	PyPI, crates.io, Go modules	supply-chain.md

Source Maturity

A source's release maturity is the maturity field of its shipped profile under crates/fetcher/profiles/; the capability catalog (docs/capability-catalog.yaml) repeats it and a test keeps the two equal. The fetcher logs a warning at startup for any enabled source that is not yet stable.

Stage	Meaning
Alpha	Code-complete but not production-validated. Behaviour and config may change. The default for a new source.
Beta	Validated against a live service; hardening in progress.
Stable	Production-ready.