Note
Access to this page requires authorization. You can try signing in or changing directories.
Access to this page requires authorization. You can try changing directories.
Important
Lakewatch is in Private Preview. To request access, reach out to your Azure Databricks account team.
Lakewatch provides an API connector to ingest security alerts from the Microsoft Graph security API, such as alerts from Microsoft Defender XDR and Microsoft Sentinel. This article covers prerequisites, configuration instructions, the connector schema, and Lakewatch parser observables.
Prerequisites
The connector requires:
- Serverless Lakeflow pipelines turned on in your workspace.
- Unity Catalog, with privileges to create or use a Unity Catalog connection. See Connection permissions.
- If your workspace's serverless network policy uses restricted access, the source's API domains must be on the policy's Allowed domains list. See Allow the source API domains in your network policy.
- A Microsoft Entra ID tenant that uses Microsoft security products, such as Microsoft Defender XDR or Microsoft Sentinel. See Microsoft's alert resource type documentation.
- Permission to register an application in Microsoft Entra ID, and the Privileged Role Administrator role to grant admin consent for Microsoft Graph application permissions. See Grant tenant-wide admin consent to an application.
Known limitations
- Slowly changing dimension (SCD) type 2 is not supported.
- The connector supports OAuth machine-to-machine authentication with a Microsoft Entra ID application and client secret. OAuth user-to-machine and basic authentication aren't supported.
- The connector supports the Microsoft global service, including Government Community Cloud (GCC), and the US Government GCC High and Department of Defense (DoD) clouds. It doesn't support Microsoft Graph operated by 21Vianet in China.
- Historical lookback can't exceed 180 days. See Choose how far back to ingest.
Configure Microsoft Graph for ingestion
The connector authenticates as a Microsoft Entra ID application that has the SecurityAlert.Read.All application permission for Microsoft Graph. For the full steps, see Configure authentication to Microsoft Graph.
- In the Microsoft Entra admin center, browse to Entra ID > App registrations and register a single-tenant application. For a GCC High or DoD tenant, use the Azure Government portal at https://portal.azure.us.
- Under Certificates & secrets, create a client secret and copy its Value. Microsoft Entra doesn't show the value again after you leave the page.
- Under API permissions, select Add a permission > Microsoft Graph > Application permissions, and add
SecurityAlert.Read.All. The connector doesn't use delegated permissions. - Select Grant admin consent for <your tenant>. Confirm that the Type column shows Application and the Status column shows Granted for <your tenant>.
- On the application's Overview page, copy the Application (client) ID and Directory (tenant) ID.
Note
Client secrets expire after at most 24 months. Before the secret expires, create a client secret on the same application and update the Client Secret on the existing Unity Catalog connection. Don't create a new connection.
Tip
When you create a connection through the Lakewatch UI, select the Microsoft cloud your tenant runs in (commercial for the global service including GCC, gcc_high, or dod), and enter the Client ID, Client Secret, and Tenant ID that you copied in the previous steps.
Next steps
After completing the configuration above, create a datasource with the Microsoft Graph connector in the Lakewatch UI. See API connectors for instructions.
Connector and Lakewatch schemas
The connector ingests the following source table from the default source schema:
| Source table | Microsoft Graph data | Built-in Lakewatch parser |
|---|---|---|
security_alerts |
Security alerts from the Microsoft Graph security API alerts_v2 endpoint |
microsoft_graph_security_alert |
The connector writes the source table to the datasource's raw destination. The built-in parser writes a separate parsed table named after the parser, unless you set a parsed destination when you configure the datasource.
Lakewatch-managed columns
The connector uses the source id field as the non-null primary key and the source lastUpdateDateTime field as the cursor for incremental ingestion, so each sync also ingests alerts that changed since the last run. Unlike connectors that synthesize a Lakewatch ID, this connector doesn't add an lw_id column. The parser assigns the parsed time column from the source firstActivityDateTime field.
For Lakewatch-created pipelines, the connector also adds these columns to the raw table, and the parsed table carries them forward:
lw_meta: Standard Lakewatch processing metadata.lw_observables: Values extracted from fields marked as security observables in the parser.
See Parser reference.
Raw connector table schema
The following schema includes the Microsoft Graph security alert fields. For field definitions, see the Microsoft Graph Logs connector reference and Microsoft's alert resource type documentation.
Schema
| Column | Data type |
|---|---|
id |
STRING |
title |
STRING |
category |
STRING |
categories |
ARRAY<STRING> |
severity |
STRING |
status |
STRING |
serviceSource |
STRING |
detectionSource |
STRING |
detectorId |
STRING |
productName |
STRING |
providerAlertId |
STRING |
incidentId |
STRING |
incidentWebUrl |
STRING |
alertWebUrl |
STRING |
alertPolicyId |
STRING |
tenantId |
STRING |
description |
STRING |
recommendedActions |
STRING |
investigationState |
STRING |
determination |
STRING |
classification |
STRING |
assignedTo |
STRING |
actorDisplayName |
STRING |
threatDisplayName |
STRING |
threatFamilyName |
STRING |
mitreTechniques |
ARRAY<STRING> |
systemTags |
ARRAY<STRING> |
createdDateTime |
TIMESTAMP |
lastUpdateDateTime |
TIMESTAMP |
firstActivityDateTime |
TIMESTAMP |
lastActivityDateTime |
TIMESTAMP |
resolvedDateTime |
TIMESTAMP |
comments |
VARIANT |
evidence |
VARIANT |
additionalData |
VARIANT |
customDetails |
VARIANT |
Parser observables
The microsoft_graph_security_alert parser writes nested objects as VARIANT and extracts the following fields into lw_observables.
Observables
| Field | Observable |
|---|---|
mitreTechniques[] |
mitre_attack_technique |
The fields in evidence depend on the type of each evidence entity, such as a device or file. The parser scans evidence and extracts values into lw_observables by matching subfield names. Matching is case-insensitive.
Evidence observables
evidence subfield name |
Observable |
|---|---|
ipAddress, ipInterfaces, lastIpAddress, lastExternalIpAddress, clusterIP, externalIPs |
ip |
dnsDomain, domainName |
domain |
hostName, deviceDnsName |
hostname |
url |
url |
md5, sha1, sha256, value |
hash |
accountName |
username |
userPrincipalName |
email |
amazonResourceId |
aws_arn |
amazonAccountId |
aws_account_id |