Skip to main content

M365 Setup for dfe-fetcher

What a cloud administrator configures so dfe-fetcher can read security and audit data from Microsoft 365 (read-only, pull-mode).

Overview​

dfe-fetcher reads M365 over two parallel surfaces, authenticating to each via the OAuth2 client_credentials flow against https://login.microsoftonline.com/<tenant>/oauth2/v2.0/token:

  • Office 365 Management Activity API (manage.office.com, scope https://manage.office.com/.default) - raw per-event audit records, fetched per content type. Backs audit_log, dlp, and exchange_audit.
  • Microsoft Graph Security (graph.microsoft.com, scope https://graph.microsoft.com/.default) - curated, post-classification alerts from security/alerts_v2. Backs alerts only.

Services and their behaviour:

ServiceSurfaceWhat it pulls
audit_logOMAPIterates content types. Default set: Audit.General, Audit.AzureActiveDirectory, Audit.Exchange, Audit.SharePoint, DLP.All. Emits one result per content type, tagged m365.audit_log.<content_type>. Override with service config content_types: [...].
exchange_auditOMAPAudit.Exchange only - per-message audit records. Tagged m365.exchange_audit.
dlpOMAPDLP.All only - raw DLP rule-match events. Tagged m365.dlp.
alertsGraphGET /v1.0/security/alerts_v2. Tagged m365.alerts.

The source is the shipped m365 profile (crates/fetcher/profiles/m365.yaml); the sources.m365 block below maps onto an instance of it at load. On the OMAP path every tick starts each content type's subscription blind (the API answers an already-enabled feed with a 400 the profile ignores), asks for the content list in steps of at most a day, follows NextPageUri, and fetches each listed blob in list order; every row carries its blob as a checkpoint item, committed once the row is acknowledged. One token per audience is minted per instance and shared by the units that need it. A 429 or 5xx is retried with backoff (honouring Retry-After), a 401, 403 or 404 ends the tick with the API's error.message, and a tick that fails does not advance the fetch window. A one-time manual subscription start is still the recommended bootstrap (see Source-Side Setup).

Prerequisites​

  • An Entra ID (Azure AD) tenant with M365 audit logging enabled in the Purview / compliance portal (the unified audit log must be turned on for OMAP to return content).
  • An Entra app registration (client_id) with a client secret, in the target tenant (tenant_id).
  • A Global Administrator (or Application Administrator) to grant admin consent.
  • az CLI logged into the target tenant, or OpenTofu with az login.

Required Permissions​

All permissions are application (app-only, not delegated) and read-only. Resource App IDs: Office 365 Management API = c5393580-f805-4401-95e8-94b7a6ef2fc2, Microsoft Graph = 00000003-0000-0000-c000-000000000000.

Used byAPIPermissionPermission IDType
audit_log, exchange_auditOffice 365 ManagementActivityFeed.Read594c1fb6-4f81-4475-ae41-0c394909246cApplication
dlpOffice 365 ManagementActivityFeed.ReadDlp4807a72c-ad38-4250-94c9-4eabfe26cd55Application
alertsMicrosoft GraphSecurityAlert.Read.All472e4a4d-bb4a-4026-98d1-0b0d74cb74a5Application

Notes:

  • ActivityFeed.Read covers the Audit.* content types. DLP.All content (sensitive-data detections) requires ActivityFeed.ReadDlp. Grant both if you fetch dlp or the default audit_log set (which includes DLP.All).
  • 472e4a4d-bb4a-4026-98d1-0b0d74cb74a5 is the application ID for SecurityAlert.Read.All (the delegated variant is a different GUID). This is the only Graph permission the current code needs - it reads alerts_v2. SecurityEvents.Read.All (the alerts_v1 permission) and Reports.Read.All are provisioned by the test module for convenience but are not required by the live code path.

Source-Side Setup​

OpenTofu (Automated)​

The test infrastructure lives in infra/test/main.tf (M365 section). It creates an app registration with the permission IDs above, a service principal, and a client secret.

cd infra/test
cp terraform.tfvars.example terraform.tfvars
az login   # must target the M365 tenant
tofu init
tofu apply
tofu output -json | python3 gen-env.py > ../../.env

The module cannot grant admin consent. After apply (requires Global Admin):

az ad app permission admin-consent --id <m365_client_id>

Manual Setup (az CLI)​

  1. Create the app registration

    az ad app create --display-name "dfe-fetcher-m365"

  2. Add the Office 365 Management API permissions

    # ActivityFeed.Read
    az ad app permission add --id <app_id> \
      --api c5393580-f805-4401-95e8-94b7a6ef2fc2 \
      --api-permissions 594c1fb6-4f81-4475-ae41-0c394909246c=Role
    # ActivityFeed.ReadDlp (for the dlp service / DLP.All content)
    az ad app permission add --id <app_id> \
      --api c5393580-f805-4401-95e8-94b7a6ef2fc2 \
      --api-permissions 4807a72c-ad38-4250-94c9-4eabfe26cd55=Role

  3. Add the Microsoft Graph permission

    az ad app permission add --id <app_id> \
      --api 00000003-0000-0000-c000-000000000000 \
      --api-permissions 472e4a4d-bb4a-4026-98d1-0b0d74cb74a5=Role

  4. Grant admin consent (Global Admin)

    az ad app permission admin-consent --id <app_id>

  5. Create a client secret

    az ad app credential reset --id <app_id> --years 1

  6. Create the service principal (if not auto-created)

    az ad sp create --id <app_id>

  7. Activate the OMAP audit subscriptions. The Office 365 Management Activity API will not return content until a subscription is started per content type. dfe-fetcher starts each subscription on every tick, but do this once to bootstrap so the very first fetch has data:

    TENANT_ID="your-tenant-uuid"
    TOKEN=$(az account get-access-token \
      --resource https://manage.office.com --query accessToken -o tsv)

    for CT in Audit.General Audit.AzureActiveDirectory Audit.Exchange Audit.SharePoint DLP.All; do
      curl -s -X POST \
        "https://manage.office.com/api/v1.0/${TENANT_ID}/activity/feed/subscriptions/start?contentType=${CT}" \
        -H "Authorization: Bearer ${TOKEN}" -H "Content-Type: application/json" -d '{}'
    done

    Subscriptions persist until explicitly stopped. Verify consent later with:

    az ad app permission list-grants --id <app_id>

dfe-fetcher Configuration​

Config File​

sources:
  m365:
    enabled: true
    tenant_id: "your-tenant-uuid"
    client_id: "your-client-uuid"
    client_secret: "your-client-secret"
    # publisher_identifier: "your-own-guid"   # keys this deployment's OMAP rate budget
    services:
      - name: audit_log
        # config:
        #   content_types: ["Audit.General", "Audit.SharePoint"]  # opt out of feeds
      - name: exchange_audit
      - name: dlp
      - name: alerts
    topic: "m365"
    # filter: 'Workload != "Exchange"'   # CEL, hot-reloaded

Recognised top-level fields: enabled, tenant_id, client_id, client_secret, credential_secret, publisher_identifier, interval_secs, services, connections, topic, filter, management_url_override, graph_url_override, token_url_override. Service names are audit_log, exchange_audit, dlp and alerts; the OMAP Audit.Exchange feed is exchange_audit. A missing tenant or secret, an unknown service name, or a content_types entry outside the five feeds is refused at load, naming sources.m365.

publisher_identifier sets the OMAP PublisherIdentifier on every Management Activity call, including the subscription start. It keys the per-publisher rate budget, so set your own GUID per deployment rather than sharing the default; it is a config key, not an environment variable.

Environment Variables​

Pattern: DFE_FETCHER_SOURCES__M365__<FIELD> (double underscores).

DFE_FETCHER_SOURCES__M365__ENABLED="true"
DFE_FETCHER_SOURCES__M365__TENANT_ID="your-tenant-uuid"
DFE_FETCHER_SOURCES__M365__CLIENT_ID="your-client-uuid"
DFE_FETCHER_SOURCES__M365__CLIENT_SECRET="your-client-secret"

Secrets Manager​

Set credential_secret to a secret-store spec. It supplies the client SECRET only and resolves to a single string, so keep client_id as the literal application id beside it. A JSON blob is not parsed.

sources:
  m365:
    enabled: true
    tenant_id: "your-tenant-uuid"
    client_id: "your-client-uuid"
    credential_secret: "vault:kv/data/dfe/m365:client_secret"
    services:
      - name: audit_log
      - name: alerts
    topic: "m365"

For multiple tenants (MSP scenario), run one dfe-fetcher instance per tenant - each with its own app registration, admin consent, activated subscriptions, and a distinct instance_id.

Verification​

  1. Health check. The health check is the management token exchange. A healthy result confirms OMAP-backed services will authenticate; Graph alerts needs the separate SecurityAlert.Read.All grant, which the health check does not exercise.

  2. Env-gated smoke tests. Live tests live in crates/fetcher/tests/e2e/smoke_remote.rs, all #[ignore]'d. They read M365_TENANT_ID, M365_CLIENT_ID and M365_CLIENT_SECRET from .env-cloud, falling back to the AZURE_* equivalents when an M365 variable is unset.

    cargo test -p dfe-fetcher --test e2e m365_ -- --ignored

  3. Common failures.

    • 403 Forbidden on every call: admin consent not granted.
    • 403 on alerts only: missing SecurityAlert.Read.All.
    • audit_log/dlp returns nothing: unified audit log not enabled in Purview, or the subscription was never started (the bootstrap step). A 404 on the content list fails that unit's tick; the next tick's subscription start re-arms it and the window has not moved.
    • DLP content missing while Audit.* works: missing ActivityFeed.ReadDlp.
    • HTTP 429: OMAP throttling - dfe-fetcher retries with backoff and Retry-After up to the profile's policy, then fails the tick without advancing; set your own publisher_identifier so the deployment has its own rate budget.

Cost​

dfe-fetcher only reads. Entra ID app registrations, Office 365 Management Activity API reads, and Microsoft Graph Security API reads are not expected to carry an additional charge, though this can depend on your M365 licensing. Confirm any cost implications against your own Microsoft agreement.

References​