Microsoft Entra ID Protection logs

Important

Lakewatch is in Private Preview. To request access, reach out to your Azure Databricks account team.

Overview

Microsoft Entra ID Protection logs record user risk. Lakewatch ingests two diagnostic categories from cloud storage using Auto Loader and schematizes them with the built-in azure_identity_protection parser:

• RiskyUsers: The current risk state of each user flagged as risky.
• UserRiskEvents: Individual risk detections, with the source IP address, location, and MITRE ATT&CK technique.

User risk events store extra context, such as the risk reasons and user agent, as a JSON-encoded string in the additionalInfo column. Parse it with from_json to query individual values. This is a cloud storage source, not an API connector.

Onboard Microsoft Entra ID Protection logs

To ingest Microsoft Entra ID Protection logs, create a cloud-storage datasource for them and attach the built-in azure_identity_protection parser so events are written to a schematized parsed table. For prerequisites, Unity Catalog access, and the full datasource setup steps, see Cloud storage.

The parser claims only events whose category is RiskyUsers or UserRiskEvents. If one storage location mixes several Azure log categories, attach every matching Azure parser to the same datasource, with the generic azure_monitor_activity parser last. See Ingest several Azure log categories from one location.

Parsed schema and observables

The built-in azure_identity_protection parser schematizes Microsoft Entra ID Protection logs into a parsed table. Every parsed table also carries the Lakewatch-managed columns:

• lw_id: A non-null string Lakewatch uses as the primary key, supporting deduplication across incremental reads.
• time: A non-null event timestamp assigned by the parser.
• lw_meta: Standard Lakewatch processing metadata.
• lw_observables: Values extracted from fields marked as security observables in the parser.

The full event payload is kept in the properties column as a VARIANT, so fields that the parser doesn't promote to their own columns remain queryable. For more information, see the Parser reference.

Parsed table schema

The azure_identity_protection parser writes the following columns (generated from the parser definition):

Column	Data type
time	TIMESTAMP
resourceId	STRING
operationName	STRING
operationVersion	STRING
category	STRING
tenantId	STRING
resultSignature	STRING
durationMs	DOUBLE
callerIpAddress	STRING
correlationId	STRING
identity	STRING
level	STRING
location	STRING
userId	STRING
userPrincipalName	STRING
userDisplayName	STRING
riskType	STRING
riskEventType	STRING
riskLevel	STRING
riskState	STRING
riskDetail	STRING
detectionSource	STRING
detectionTimingType	STRING
activity	STRING
mitreTechniqueId	STRING
additionalInfo	STRING
ipAddress	STRING
locationCity	STRING
locationCountryOrRegion	STRING
locationLatitude	DOUBLE
locationLongitude	DOUBLE
properties	VARIANT

Parser observables

The parser extracts the following fields into lw_observables (generated from the parser definition):

Field	Observable
callerIpAddress	ip
correlationId	trace_id
userId	actor_id
userPrincipalName	username
userPrincipalName	email
userDisplayName	username
mitreTechniqueId	mitre_attack_technique
additionalInfo	mitre_attack_technique
ipAddress	ip

The following VARIANT fields are scanned for observables by matching nested key names; the observable types extracted from each:

Field	Observable types scanned
properties	trace_id