Microsoft 365 audit logs

Important

Lakewatch is in Private Preview. To request access, reach out to your Azure Databricks account team.

Lakewatch provides an API connector to ingest audit logs from Microsoft 365. This article covers prerequisites, configuration instructions, connector schemas, and Lakewatch parser observables.

Prerequisites

The connector requires:

  1. Serverless Lakeflow Spark Declarative Pipelines turned on in your workspace.
  2. Unity Catalog, with privileges to create or use a Unity Catalog connection. See Connection permissions.
  3. Office 365 unified auditing enabled for your organization.
  4. Permission to register an application in Microsoft Entra ID.
  5. Permission to grant tenant-wide admin consent for Office 365 Management APIs.

Known limitations

  • SCD type 2 is not supported, since Microsoft 365 Audit Logs are append-only.
  • As per the Microsoft 365 Management API documentation, audit logs can be retrieved only after a subscription to fetch the content blobs has been created and activated. The connector attempts to create and activate the subscription at the start of ingestion. Therefore, if a subscription was not already enabled, the connector ingests only those audit records generated after the subscription becomes active. We also recommend not disabling the subscription midway to avoid any potential data loss.
  • The Microsoft 365 Management API provides access to audit logs only for the most recent seven days from the creation of subscription. Hence if the interval between two consecutive pipeline executions exceeds seven days, the connector ingests only the logs corresponding to past week. To ensure no data loss, schedule the pipeline to run at intervals much shorter than 7 days.
  • Historical lookback can't exceed the 7 days Microsoft 365 retains audit logs. See Choose how far back to ingest.

Configure Microsoft 365 for ingestion

In this step, you register an application in the Azure portal to enable authentication from Databricks.

Register the application

  1. Go to the Azure portal and search for Microsoft Entra ID.
  2. In the left pane, go to Manage > App registrations, and click New registration.
  3. On the Register an application page:
    1. Enter a user-facing display name for your application.
    2. In Supported account types, select Single tenant only - <your tenant name>.
    3. Leave Redirect URI blank.
    4. Click Register.

Create a client secret

  1. In the left pane of the registered application page, go to Manage > Certificates & secrets, and click New client secret.
  2. In the pane that opens, enter a description for the secret, select 24 months for Expires, and click Add.
  3. Note the credential Value, not the Secret ID. You use this value in the next step.

Configure API permissions

  1. Add delegated permissions:
    1. In the left pane, go to Manage > API permissions, and click Add a permission.
    2. In the Request API permissions pane, click Office 365 Management APIs.
    3. Click Delegated permissions.
    4. Select ActivityFeed.Read, ActivityFeed.ReadDlp, and ServiceHealth.Read.
    5. Click Add permissions.
  2. Repeat the permission steps for application permissions:
    1. Click Add a permission.
    2. In the Request API permissions pane, click Office 365 Management APIs.
    3. Click Application permissions.
    4. Select ActivityFeed.Read, ActivityFeed.ReadDlp, and ServiceHealth.Read.
    5. Click Add permissions.
  3. On the API permissions page, click Grant admin consent for <your organization>, next to Add a permission, and click Yes in the confirmation popup.

Copy the application IDs

  1. In the left pane, go to Overview.
  2. Note down the Application (client) ID and Directory (tenant) ID; you'll use these in the next step.

Tip

When creating a connection through the Lakewatch UI, you need the Client ID, Client Secret, and Tenant ID gathered above.

Next steps

After completing the configuration above, create a datasource with the Microsoft 365 connector in the Lakewatch UI. See API connectors for instructions.


Connector and Lakewatch schemas

The Microsoft 365 Audit Logs connector supports five source tables in the default source schema. Microsoft can add fields and event types to these payloads. For field definitions, see the Microsoft 365 Management Activity API schema.

Source table Microsoft content type Built-in Lakewatch parser
audit_azure_active_directory Audit.AzureActiveDirectory microsoft365_audit_azure_active_directory
audit_exchange Audit.Exchange microsoft365_audit_exchange
audit_sharepoint Audit.SharePoint microsoft365_audit_sharepoint
audit_general Audit.General microsoft365_audit_general
dlp_all DLP.All microsoft365_dlp_all

The connector writes each selected source table to the datasource's raw destination. Each source table has a registered built-in parser that writes a separate parsed table named after the parser, unless you set a parsed destination when you configure the datasource.

Lakewatch-managed columns

The connector adds the following columns to each raw source row:

  • lw_id: A non-null string that Lakewatch uses as the primary key. The connector derives a stable, time-sortable ID from the Microsoft CreationTime and Id fields. The ID supports deduplication across incremental syncs.
  • time: A non-null UTC timestamp parsed from CreationTime. The connector uses time as the cursor field for incremental ingestion.

Parsed tables carry lw_id forward. The parser also assigns the parsed time column and adds:

  • lw_meta: Standard Lakewatch processing metadata.
  • lw_observables: Values extracted from fields marked as security observables in the parser.

For more information, see Parser reference.

Raw connector table schemas

The following schemas combine the Microsoft 365 event fields with lw_id and time.


audit_azure_active_directory

Captures Azure Active Directory audit events including user sign-ins, app access, and directory changes. For more information about the schema, refer to Microsoft 365 Management API docs.

Schema
Field Data type
lw_id string
time timestamp
Id string
RecordType integer
CreationTime timestamp
Operation string
OrganizationId string
UserType integer
UserKey string
Workload string
Version integer
ResultStatus string
ObjectId string
UserId string
ClientIP string
Scope string
AppAccessContext struct
AppAccessContext.AADSessionId string
AppAccessContext.APIId string
AppAccessContext.ClientAppId string
AppAccessContext.ClientAppName string
AppAccessContext.CorrelationId string
AzureActiveDirectoryEventType integer
ExtendedProperties array<struct>
ExtendedProperties.Name string
ExtendedProperties.Value string
ModifiedProperties array<map<string, variant>>
DeviceProperties array<struct>
DeviceProperties.Name string
DeviceProperties.Value string
Application string
Client string
LoginStatus integer
UserDomain string
Actor array<struct>
Actor.ID string
Actor.Type integer
ActorContextId string
ActorIpAddress string
InterSystemsId string
IntraSystemId string
SupportTicketId string
Target array<struct>
Target.ID string
Target.Type integer
TargetContextId string
ApplicationId string
ErrorNumber string
LogonError string

audit_exchange

Captures Exchange mailbox audit events including email access, folder operations, and item movements. For more information about the schema, refer to Microsoft 365 Management API docs.

Schema
Field Data type
lw_id string
time timestamp
Id string
RecordType integer
CreationTime timestamp
Operation string
OrganizationId string
UserType integer
UserKey string
Workload string
Version integer
ResultStatus string
ObjectId string
UserId string
ClientIP string
Scope string
AppAccessContext struct
AppAccessContext.AADSessionId string
AppAccessContext.APIId string
AppAccessContext.ClientAppId string
AppAccessContext.ClientAppName string
AppAccessContext.CorrelationId string
ModifiedObjectResolvedName string
Parameters array<struct>
Parameters.Name string
Parameters.Value string
ModifiedProperties array<map<string, variant>>
ExternalAccess boolean
OriginatingServer string
OrganizationName string
LogonType integer
InternalLogonType integer
MailboxGuid string
MailboxOwnerUPN string
MailboxOwnerSid string
MailboxOwnerMasterAccountSid string
LogonUserSid string
LogonUserDisplayName string
ClientInfoString string
ClientIPAddress string
ClientMachineName string
ClientProcessName string
ClientVersion string
Folder struct
Folder.Id string
Folder.Path string
Folder.FolderItems array<struct>
Folder.FolderItems.Id string
Folder.FolderItems.ImmutableId string
Folder.FolderItems.InternetMessageId string
Folder.FolderItems.CreationTime timestamp
Folder.FolderItems.Subject string
Folder.FolderItems.SizeInBytes long
Folder.FolderItems.Sensitivity string
Folder.FolderItems.ClientRequestId string
Folder.FolderItems.Teams variant
CrossMailboxOperations boolean
DestMailboxId string
DestMailboxOwnerUPN string
DestMailboxOwnerSid string
DestMailboxOwnerMasterAccountSid string
DestFolder struct
DestFolder.Id string
DestFolder.Path string
DestFolder.FolderItems array<struct>
DestFolder.FolderItems.Id string
DestFolder.FolderItems.ImmutableId string
DestFolder.FolderItems.InternetMessageId string
DestFolder.FolderItems.CreationTime timestamp
DestFolder.FolderItems.Subject string
DestFolder.FolderItems.SizeInBytes long
DestFolder.FolderItems.Sensitivity string
DestFolder.FolderItems.ClientRequestId string
DestFolder.FolderItems.Teams variant
Folders array<struct>
Folders.Id string
Folders.Path string
Folders.FolderItems array<struct>
Folders.FolderItems.Id string
Folders.FolderItems.ImmutableId string
Folders.FolderItems.InternetMessageId string
Folders.FolderItems.CreationTime timestamp
Folders.FolderItems.Subject string
Folders.FolderItems.SizeInBytes long
Folders.FolderItems.Sensitivity string
Folders.FolderItems.ClientRequestId string
Folders.FolderItems.Teams variant
AffectedItems array<struct>
AffectedItems.Id string
AffectedItems.Subject string
AffectedItems.ParentFolder struct
AffectedItems.ParentFolder.Id string
AffectedItems.ParentFolder.Path string
AffectedItems.ParentFolder.FolderItems array<struct>
AffectedItems.ParentFolder.FolderItems.Id string
AffectedItems.ParentFolder.FolderItems.ImmutableId string
AffectedItems.ParentFolder.FolderItems.InternetMessageId string
AffectedItems.ParentFolder.FolderItems.CreationTime timestamp
AffectedItems.ParentFolder.FolderItems.Subject string
AffectedItems.ParentFolder.FolderItems.SizeInBytes long
AffectedItems.ParentFolder.FolderItems.Sensitivity string
AffectedItems.ParentFolder.FolderItems.ClientRequestId string
AffectedItems.ParentFolder.FolderItems.Teams variant
AffectedItems.Attachments string
Item struct
Item.Id string
Item.Subject string
Item.ParentFolder struct
Item.ParentFolder.Id string
Item.ParentFolder.Path string
Item.ParentFolder.FolderItems array<struct>
Item.ParentFolder.FolderItems.Id string
Item.ParentFolder.FolderItems.ImmutableId string
Item.ParentFolder.FolderItems.InternetMessageId string
Item.ParentFolder.FolderItems.CreationTime timestamp
Item.ParentFolder.FolderItems.Subject string
Item.ParentFolder.FolderItems.SizeInBytes long
Item.ParentFolder.FolderItems.Sensitivity string
Item.ParentFolder.FolderItems.ClientRequestId string
Item.ParentFolder.FolderItems.Teams variant
Item.Attachments string
SendAsUserSmtp string
SendAsUserMailboxGuid string
SendOnBehalfOfUserSmtp string
SendOnBehalfOfUserMailboxGuid string

audit_sharepoint

Captures SharePoint audit events including file access, list operations, and sharing activities. For more information on the schema, refer to Microsoft 365 API documentation.

Schema
Field Data type
lw_id string
time timestamp
Id string
RecordType integer
CreationTime timestamp
Operation string
OrganizationId string
UserType integer
UserKey string
Workload string
Version integer
ResultStatus string
ObjectId string
UserId string
ClientIP string
Scope string
AppAccessContext struct
AppAccessContext.AADSessionId string
AppAccessContext.APIId string
AppAccessContext.ClientAppId string
AppAccessContext.ClientAppName string
AppAccessContext.CorrelationId string
Site string
ItemType string
EventSource string
SourceName string
UserAgent string
MachineDomainInfo string
MachineId string
ListItemUniqueId string
ListId string
ApplicationId string
ApplicationDisplayName string
IsWorkflow boolean
SiteUrl string
SourceRelativeUrl string
SourceFileName string
SourceFileExtension string
DestinationRelativeUrl string
DestinationFileName string
DestinationFileExtension string
UserSharedWith string
SharingType string
SourceLabel string
DestinationLabel string
SensitivityLabelOwnerEmail string
SensitivityLabelId string
ListTitle string
ListName string
ListUrl string
ListBaseType string
ListBaseTemplateType string
IsHiddenList boolean
IsDocLib boolean
TargetUserOrGroupName string
TargetUserOrGroupType string
UniqueSharingId string
CustomEvent string
EventData string
ModifiedProperties array<map<string, variant>>

audit_general

Captures miscellaneous M365 audit events not covered by other workload-specific schemas, with a flexible variant payload. For more information on schema, refer to Microsoft 365 Management API documentation.

Schema
Field Data type
lw_id string
time timestamp
Id string
RecordType integer
CreationTime timestamp
Operation string
OrganizationId string
UserType integer
UserKey string
Workload string
Version integer
ResultStatus string
ObjectId string
UserId string
ClientIP string
Scope string
AppAccessContext struct
AppAccessContext.AADSessionId string
AppAccessContext.APIId string
AppAccessContext.ClientAppId string
AppAccessContext.ClientAppName string
AppAccessContext.CorrelationId string
payload variant

dlp_all

Captures Data Loss Prevention (DLP) policy match events across Exchange, SharePoint, and endpoint workloads. For more information on schema, refer to Microsoft 365 Management API docs.

Schema
Field Data type
lw_id string
time timestamp
Id string
RecordType integer
CreationTime timestamp
Operation string
OrganizationId string
UserType integer
UserKey string
Workload string
Version integer
ResultStatus string
ObjectId string
UserId string
ClientIP string
Scope string
AppAccessContext struct
AppAccessContext.AADSessionId string
AppAccessContext.APIId string
AppAccessContext.ClientAppId string
AppAccessContext.ClientAppName string
AppAccessContext.CorrelationId string
SharePointMetaData struct
SharePointMetaData.From string
SharePointMetaData.itemCreationTime timestamp
SharePointMetaData.SiteCollectionGuid string
SharePointMetaData.SiteCollectionUrl string
SharePointMetaData.FileName string
SharePointMetaData.FileOwner string
SharePointMetaData.FilePathUrl string
SharePointMetaData.DocumentLastModifier string
SharePointMetaData.DocumentSharer string
SharePointMetaData.UniqueId string
SharePointMetaData.LastModifiedTime timestamp
SharePointMetaData.IsViewableByExternalUsers boolean
ExchangeMetaData struct
ExchangeMetaData.MessageID string
ExchangeMetaData.From string
ExchangeMetaData.To array<string>
ExchangeMetaData.CC array<string>
ExchangeMetaData.BCC array<string>
ExchangeMetaData.Subject string
ExchangeMetaData.Sent timestamp
ExchangeMetaData.RecipientCount integer
EndPointMetaData struct
EndPointMetaData.SensitiveInformation array<struct>
EndPointMetaData.SensitiveInformation.Confidence integer
EndPointMetaData.SensitiveInformation.Count integer
EndPointMetaData.SensitiveInformation.Location string
EndPointMetaData.SensitiveInformation.SensitiveType string
EndPointMetaData.SensitiveInformation.SensitiveInformationDetections struct
EndPointMetaData.SensitiveInformation.SensitiveInformationDetections.DetectedValues array<struct>
EndPointMetaData.SensitiveInformation.SensitiveInformationDetections.DetectedValues.Name string
EndPointMetaData.SensitiveInformation.SensitiveInformationDetections.DetectedValues.Value string
EndPointMetaData.SensitiveInformation.SensitiveInformationDetections.ResultsTruncated boolean
EndPointMetaData.SensitiveInformation.SensitiveInformationDetailedClassificationAttributes array<struct>
EndPointMetaData.SensitiveInformation.SensitiveInformationDetailedClassificationAttributes.Confidence integer
EndPointMetaData.SensitiveInformation.SensitiveInformationDetailedClassificationAttributes.Count integer
EndPointMetaData.SensitiveInformation.SensitiveInformationDetailedClassificationAttributes.IsMatch boolean
EndPointMetaData.SensitiveInformation.SensitiveInformationTypeName string
EndPointMetaData.SensitiveInformation.UniqueCount integer
EndPointMetaData.EnforcementMode string
EndPointMetaData.FileExtension string
EndPointMetaData.FileType string
EndPointMetaData.DeviceName string
ExceptionInfo string
PolicyDetails array<struct>
PolicyDetails.PolicyId string
PolicyDetails.PolicyName string
PolicyDetails.Rules array<struct>
PolicyDetails.Rules.RuleId string
PolicyDetails.Rules.RuleName string
PolicyDetails.Rules.Actions array<string>
PolicyDetails.Rules.OverriddenActions array<string>
PolicyDetails.Rules.Severity string
PolicyDetails.Rules.RuleMode string
PolicyDetails.Rules.ConditionsMatched struct
PolicyDetails.Rules.ConditionsMatched.SensitiveInformation array<struct>
PolicyDetails.Rules.ConditionsMatched.SensitiveInformation.Confidence integer
PolicyDetails.Rules.ConditionsMatched.SensitiveInformation.Count integer
PolicyDetails.Rules.ConditionsMatched.SensitiveInformation.Location string
PolicyDetails.Rules.ConditionsMatched.SensitiveInformation.SensitiveType string
PolicyDetails.Rules.ConditionsMatched.SensitiveInformation.SensitiveInformationDetections struct
PolicyDetails.Rules.ConditionsMatched.SensitiveInformation.SensitiveInformationDetections.DetectedValues array<struct>
PolicyDetails.Rules.ConditionsMatched.SensitiveInformation.SensitiveInformationDetections.DetectedValues.Name string
PolicyDetails.Rules.ConditionsMatched.SensitiveInformation.SensitiveInformationDetections.DetectedValues.Value string
PolicyDetails.Rules.ConditionsMatched.SensitiveInformation.SensitiveInformationDetections.ResultsTruncated boolean
PolicyDetails.Rules.ConditionsMatched.SensitiveInformation.SensitiveInformationDetailedClassificationAttributes array<struct>
PolicyDetails.Rules.ConditionsMatched.SensitiveInformation.SensitiveInformationDetailedClassificationAttributes.Confidence integer
PolicyDetails.Rules.ConditionsMatched.SensitiveInformation.SensitiveInformationDetailedClassificationAttributes.Count integer
PolicyDetails.Rules.ConditionsMatched.SensitiveInformation.SensitiveInformationDetailedClassificationAttributes.IsMatch boolean
PolicyDetails.Rules.ConditionsMatched.SensitiveInformation.SensitiveInformationTypeName string
PolicyDetails.Rules.ConditionsMatched.SensitiveInformation.UniqueCount integer
PolicyDetails.Rules.ConditionsMatched.DocumentProperties array<struct>
PolicyDetails.Rules.ConditionsMatched.DocumentProperties.Name string
PolicyDetails.Rules.ConditionsMatched.DocumentProperties.Value string
PolicyDetails.Rules.ConditionsMatched.OtherConditions array<struct>
PolicyDetails.Rules.ConditionsMatched.OtherConditions.Name string
PolicyDetails.Rules.ConditionsMatched.OtherConditions.Value string
SensitiveInfoDetectionIsIncluded boolean

Parser observables

Each built-in parser extracts the fields listed below into lw_observables.

microsoft365_audit_azure_active_directory

Observables
Field Observable
UserKey username
UserId username
ClientIP ip
ClientIP net_addr
AppAccessContext.CorrelationId trace_id
Actor[].ID username
ActorIpAddress ip
Target[].ID username

microsoft365_audit_exchange

Observables
Field Observable
UserKey username
UserId username
ClientIP ip
ClientIP net_addr
AppAccessContext.CorrelationId trace_id
MailboxOwnerUPN username
MailboxOwnerUPN email
MailboxOwnerSid actor_id
MailboxOwnerMasterAccountSid actor_id
LogonUserSid actor_id
ClientIPAddress ip
ClientIPAddress net_addr
ClientMachineName hostname
DestMailboxOwnerUPN username
DestMailboxOwnerUPN email
DestMailboxOwnerSid actor_id
DestMailboxOwnerMasterAccountSid actor_id
SendAsUserSmtp email
SendOnBehalfOfUserSmtp email

microsoft365_audit_sharepoint

Observables
Field Observable
UserKey username
UserId username
ClientIP ip
ClientIP net_addr
AppAccessContext.CorrelationId trace_id
SiteUrl url
SourceRelativeUrl url
DestinationRelativeUrl url
UserSharedWith username
SensitivityLabelOwnerEmail email
ListUrl url
TargetUserOrGroupName email

microsoft365_audit_general

Observables
Field Observable
UserKey username
UserId username
ClientIP ip
ClientIP net_addr
AppAccessContext.CorrelationId trace_id

The Audit.General payload holds event keys that have no column in the common schema, so its shape varies by workload. The parser also scans the payload field and extracts ip, domain, hostname, url, email, username, actor_id, and trace_id values into lw_observables by matching subfield names.

microsoft365_dlp_all

Observables
Field Observable
UserKey username
UserId username
ClientIP ip
ClientIP net_addr
AppAccessContext.CorrelationId trace_id
SharePointMetaData.From username
SharePointMetaData.From email
SharePointMetaData.SiteCollectionUrl url
SharePointMetaData.FileOwner username
SharePointMetaData.FileOwner email
SharePointMetaData.FilePathUrl url
SharePointMetaData.DocumentLastModifier username
SharePointMetaData.DocumentLastModifier email
SharePointMetaData.DocumentSharer username
SharePointMetaData.DocumentSharer email
ExchangeMetaData.From username
ExchangeMetaData.From email
ExchangeMetaData.To[] email
ExchangeMetaData.CC[] email
ExchangeMetaData.BCC[] email
EndPointMetaData.DeviceName hostname

Frequently asked questions

I am unable to find my subscription plan to be used while making UC connection

If the subscription plan isn't available, use a tenant ID lookup website like https://tenantidlookup.com/ and enter your tenant ID. Look at the Tenant Region Sub Scope field. If it's empty, you're on the Enterprise plan; otherwise, it shows the plan.


Troubleshooting

I'm not able to see any records ingested in the first sync

According to Microsoft 365 Management API documentation, it can take up to 12 hours for the first content blobs to become available after a subscription is created. Therefore, if the subscription was not previously enabled, there may be a delay of up to 12 hours before audit log data becomes available for ingestion.