# M365 Setup for dfe-fetcher > What a cloud administrator configures so dfe-fetcher can read security and > audit data from Microsoft 365 (read-only, pull-mode). ## Overview dfe-fetcher reads M365 over two parallel surfaces, authenticating to each via the OAuth2 client_credentials flow against `https://login.microsoftonline.com//oauth2/v2.0/token`: - **Office 365 Management Activity API** (`manage.office.com`, scope `https://manage.office.com/.default`) - raw per-event audit records, fetched per content type. Backs `audit_log`, `dlp`, and `exchange_audit`. - **Microsoft Graph Security** (`graph.microsoft.com`, scope `https://graph.microsoft.com/.default`) - curated, post-classification alerts from `security/alerts_v2`. Backs `alerts` only. Services and their behaviour: | Service | Surface | What it pulls | |---------|---------|---------------| | `audit_log` | OMAP | Iterates content types. Default set: `Audit.General`, `Audit.AzureActiveDirectory`, `Audit.Exchange`, `Audit.SharePoint`, `DLP.All`. Emits one result per content type, tagged `m365.audit_log.`. Override with service config `content_types: [...]`. | | `exchange_audit` | OMAP | `Audit.Exchange` only - per-message audit records. Tagged `m365.exchange_audit`. | | `dlp` | OMAP | `DLP.All` only - raw DLP rule-match events. Tagged `m365.dlp`. | | `alerts` | Graph | `GET /v1.0/security/alerts_v2`. Tagged `m365.alerts`. | The source is the shipped `m365` profile (`crates/fetcher/profiles/m365.yaml`); the `sources.m365` block below maps onto an instance of it at load. On the OMAP path every tick starts each content type's subscription blind (the API answers an already-enabled feed with a 400 the profile ignores), asks for the content list in steps of at most a day, follows `NextPageUri`, and fetches each listed blob in list order; every row carries its blob as a checkpoint item, committed once the row is acknowledged. One token per audience is minted per instance and shared by the units that need it. A 429 or 5xx is retried with backoff (honouring `Retry-After`), a 401, 403 or 404 ends the tick with the API's `error.message`, and a tick that fails does not advance the fetch window. A one-time manual subscription start is still the recommended bootstrap (see Source-Side Setup). ## Prerequisites - An Entra ID (Azure AD) tenant with M365 audit logging enabled in the Purview / compliance portal (the unified audit log must be turned on for OMAP to return content). - An Entra app registration (`client_id`) with a client secret, in the target tenant (`tenant_id`). - A Global Administrator (or Application Administrator) to grant admin consent. - `az` CLI logged into the target tenant, or OpenTofu with `az login`. ## Required Permissions All permissions are **application** (app-only, not delegated) and **read-only**. Resource App IDs: Office 365 Management API = `c5393580-f805-4401-95e8-94b7a6ef2fc2`, Microsoft Graph = `00000003-0000-0000-c000-000000000000`. | Used by | API | Permission | Permission ID | Type | |---------|-----|------------|---------------|------| | `audit_log`, `exchange_audit` | Office 365 Management | `ActivityFeed.Read` | `594c1fb6-4f81-4475-ae41-0c394909246c` | Application | | `dlp` | Office 365 Management | `ActivityFeed.ReadDlp` | `4807a72c-ad38-4250-94c9-4eabfe26cd55` | Application | | `alerts` | Microsoft Graph | `SecurityAlert.Read.All` | `472e4a4d-bb4a-4026-98d1-0b0d74cb74a5` | Application | **Notes:** - `ActivityFeed.Read` covers the `Audit.*` content types. `DLP.All` content (sensitive-data detections) requires `ActivityFeed.ReadDlp`. Grant both if you fetch `dlp` or the default `audit_log` set (which includes `DLP.All`). - `472e4a4d-bb4a-4026-98d1-0b0d74cb74a5` is the **application** ID for `SecurityAlert.Read.All` (the delegated variant is a different GUID). This is the only Graph permission the current code needs - it reads `alerts_v2`. `SecurityEvents.Read.All` (the `alerts_v1` permission) and `Reports.Read.All` are provisioned by the test module for convenience but are not required by the live code path. ## Source-Side Setup ### OpenTofu (Automated) The test infrastructure lives in [`infra/test/main.tf`](https://github.com/hyperi-io/dfe-fetcher/blob/v1.4.54/infra/test/main.tf) (M365 section). It creates an app registration with the permission IDs above, a service principal, and a client secret. ```bash cd infra/test cp terraform.tfvars.example terraform.tfvars az login # must target the M365 tenant tofu init tofu apply tofu output -json | python3 gen-env.py > ../../.env ``` The module cannot grant admin consent. After apply (requires Global Admin): ```bash az ad app permission admin-consent --id ``` ### Manual Setup (az CLI) 1. **Create the app registration** ```bash az ad app create --display-name "dfe-fetcher-m365" ``` 2. **Add the Office 365 Management API permissions** ```bash # ActivityFeed.Read az ad app permission add --id \ --api c5393580-f805-4401-95e8-94b7a6ef2fc2 \ --api-permissions 594c1fb6-4f81-4475-ae41-0c394909246c=Role # ActivityFeed.ReadDlp (for the dlp service / DLP.All content) az ad app permission add --id \ --api c5393580-f805-4401-95e8-94b7a6ef2fc2 \ --api-permissions 4807a72c-ad38-4250-94c9-4eabfe26cd55=Role ``` 3. **Add the Microsoft Graph permission** ```bash az ad app permission add --id \ --api 00000003-0000-0000-c000-000000000000 \ --api-permissions 472e4a4d-bb4a-4026-98d1-0b0d74cb74a5=Role ``` 4. **Grant admin consent** (Global Admin) ```bash az ad app permission admin-consent --id ``` 5. **Create a client secret** ```bash az ad app credential reset --id --years 1 ``` 6. **Create the service principal** (if not auto-created) ```bash az ad sp create --id ``` 7. **Activate the OMAP audit subscriptions.** The Office 365 Management Activity API will not return content until a subscription is started per content type. dfe-fetcher starts each subscription on every tick, but do this once to bootstrap so the very first fetch has data: ```bash TENANT_ID="your-tenant-uuid" TOKEN=$(az account get-access-token \ --resource https://manage.office.com --query accessToken -o tsv) for CT in Audit.General Audit.AzureActiveDirectory Audit.Exchange Audit.SharePoint DLP.All; do curl -s -X POST \ "https://manage.office.com/api/v1.0/${TENANT_ID}/activity/feed/subscriptions/start?contentType=${CT}" \ -H "Authorization: Bearer ${TOKEN}" -H "Content-Type: application/json" -d '{}' done ``` Subscriptions persist until explicitly stopped. Verify consent later with: ```bash az ad app permission list-grants --id ``` ## dfe-fetcher Configuration ### Config File ```yaml sources: m365: enabled: true tenant_id: "your-tenant-uuid" client_id: "your-client-uuid" client_secret: "your-client-secret" # publisher_identifier: "your-own-guid" # keys this deployment's OMAP rate budget services: - name: audit_log # config: # content_types: ["Audit.General", "Audit.SharePoint"] # opt out of feeds - name: exchange_audit - name: dlp - name: alerts topic: "m365" # filter: 'Workload != "Exchange"' # CEL, hot-reloaded ``` Recognised top-level fields: `enabled`, `tenant_id`, `client_id`, `client_secret`, `credential_secret`, `publisher_identifier`, `interval_secs`, `services`, `connections`, `topic`, `filter`, `management_url_override`, `graph_url_override`, `token_url_override`. Service names are `audit_log`, `exchange_audit`, `dlp` and `alerts`; the OMAP `Audit.Exchange` feed is `exchange_audit`. A missing tenant or secret, an unknown service name, or a `content_types` entry outside the five feeds is refused at load, naming `sources.m365`. `publisher_identifier` sets the OMAP `PublisherIdentifier` on every Management Activity call, including the subscription start. It keys the per-publisher rate budget, so set your own GUID per deployment rather than sharing the default; it is a config key, not an environment variable. ### Environment Variables Pattern: `DFE_FETCHER_SOURCES__M365__` (double underscores). ```bash DFE_FETCHER_SOURCES__M365__ENABLED="true" DFE_FETCHER_SOURCES__M365__TENANT_ID="your-tenant-uuid" DFE_FETCHER_SOURCES__M365__CLIENT_ID="your-client-uuid" DFE_FETCHER_SOURCES__M365__CLIENT_SECRET="your-client-secret" ``` ### Secrets Manager Set `credential_secret` to a secret-store spec. It supplies the client SECRET only and resolves to a single string, so keep `client_id` as the literal application id beside it. A JSON blob is not parsed. ```yaml sources: m365: enabled: true tenant_id: "your-tenant-uuid" client_id: "your-client-uuid" credential_secret: "vault:kv/data/dfe/m365:client_secret" services: - name: audit_log - name: alerts topic: "m365" ``` For multiple tenants (MSP scenario), run one dfe-fetcher instance per tenant - each with its own app registration, admin consent, activated subscriptions, and a distinct `instance_id`. ## Verification 1. **Health check.** The health check is the **management** token exchange. A healthy result confirms OMAP-backed services will authenticate; Graph `alerts` needs the separate `SecurityAlert.Read.All` grant, which the health check does not exercise. 2. **Env-gated smoke tests.** Live tests live in [`crates/fetcher/tests/e2e/smoke_remote.rs`](https://github.com/hyperi-io/dfe-fetcher/blob/v1.4.54/crates/fetcher/tests/e2e/smoke_remote.rs), all `#[ignore]`'d. They read `M365_TENANT_ID`, `M365_CLIENT_ID` and `M365_CLIENT_SECRET` from `.env-cloud`, falling back to the `AZURE_*` equivalents when an M365 variable is unset. ```bash cargo test -p dfe-fetcher --test e2e m365_ -- --ignored ``` 3. **Common failures.** - `403 Forbidden` on every call: admin consent not granted. - `403` on `alerts` only: missing `SecurityAlert.Read.All`. - `audit_log`/`dlp` returns nothing: unified audit log not enabled in Purview, or the subscription was never started (the bootstrap step). A `404` on the content list fails that unit's tick; the next tick's subscription start re-arms it and the window has not moved. - DLP content missing while `Audit.*` works: missing `ActivityFeed.ReadDlp`. - HTTP 429: OMAP throttling - dfe-fetcher retries with backoff and `Retry-After` up to the profile's policy, then fails the tick without advancing; set your own `publisher_identifier` so the deployment has its own rate budget. ## Cost dfe-fetcher only reads. Entra ID app registrations, Office 365 Management Activity API reads, and Microsoft Graph Security API reads are not expected to carry an additional charge, though this can depend on your M365 licensing. Confirm any cost implications against your own Microsoft agreement. ## References - Office 365 Management Activity API reference: - Get started with Office 365 Management APIs (permissions, ActivityFeed.Read/ReadDlp): - Microsoft Graph permissions reference: - Graph security alerts_v2 (list): - OAuth2 client credentials (Microsoft identity platform):