Ingest Data to Microsoft Sentinel Data Lake

Ingest sample IdentityDrift telemetry and supporting tables into the Sentinel data lake. You can query this data in the subsequent articles in this series.

This article is part of a series that walks you through ingesting test data to the data lake, developing an agent, and publishing the example IdentityDrift platform solution. The solution includes a Security Copilot agent that correlates identity, endpoint, and cloud security signals to automate threat investigations.

Typical production approach for ISVs

ISVs should use the following approach to ingest data into Sentinel:

  1. Develop a Data Connector
    • Create a Sentinel data connector using the Sentinel Connector Builder Agent in VS Code with GitHub Copilot. This AI-assisted approach reduces development time from weeks to hours and uses natural language prompts to build polling configurations, data collection rules, connector definitions, and schemas.
    • For pull-based data sources, use Codeless Connector Framework (CCF).
    • For push-based data sources, use CCF Push.
  2. Ingest data to data lake Stream telemetry through the connector into Sentinel's data lake
  3. Use Sentinel platform Use Sentinel platform capabilities for data exploration, Graph, MCP server, and Security Copilot agents.

Alternative: If a connector already exists for your data source, see the Sentinel Data Connectors Reference to enable it and start ingesting the data.

Lab approach: Ingest sample data via KQL Jobs

Use KQL Jobs to populate data lake with sample records. This approach:

  • Mirrors the actual data schema Sample data adheres to the same schema as connector‑ingested data, letting you populate the data lake and test agent logic immediately.
  • Represents realistic attack scenarios Mock data reproduces the attack patterns and correlations you'd investigate in production.

Prerequisites

Required permissions

  • You must have the Security Administrator or Security Operator role.

Create a KQL job

To create a new KQL job, follow these steps:

  1. In the Defender portal, navigate to Sentinel > Data lake exploration > Jobs.

  2. Select Create job, then Create a new KQL job

  3. Job name - Enter a unique job name, for example CommonSecurity_ID_logs1

  4. Job Description - Provide context and purpose of the job, for example Generate logs for IdentityDrift vendor to ingest to CommonSecurity_ID_logs1

  5. Select workspace - Select the destination workspace

  6. Under "Create a new table" and enter a table name, for example CommonSecurity_ID. Tables created by KQL jobs automatically have the suffix _KQL_CL appended.

  7. Select Next

  8. Copy the KQL query from the relevant Data ingest KQL tab into the query field.

    Tip

    If the Next button is greyed out, you must choose a workspace using the Selected Workspace button.

  9. Select Next to proceed to the scheduling page.

Schedule the job

To set a run schedule for the job:

  1. On the Schedule the query job page, select your execution option:
    • Scheduled job - Job runs on a schedule you specify and ingest sample data periodically.

Schedule Job option to have recurring records added to the table.

For scheduled jobs:

  1. Select Schedule option
  2. Enter the following details:
    • Repeat frequency - Select: By minute, Hourly, Daily, Weekly, or Monthly
    • Repeat every - How often the job runs based on selected frequency
    • From date - Select date and enter time for the job to start.
    • To date - Select when the schedule finishes. Select Set job to run indefinitely to continue without end date

Review and submit

To review and submit the job:

  1. Select Next to review job details.
  2. Review all settings.
  3. Select Submit to create the job.

Data ingest KQL

To ingest data into the SigninLogs, AADRiskyUsers, CommonSecurity_ID_Logs, DeviceProcessEvents, and SecurityAlerts tables, repeat the steps to create a KQL job, using the KQL queries in each tab below.

Use the following query in your KQL job to ingest sample risky user data into the AADRiskyUsers table in the Sentinel data lake.

datatable(
    TimeGenerated:datetime,
    CorrelationId:string,
    Id:string,
    IsDeleted:bool,
    IsProcessing:bool,
    OperationName:string,
    RiskDetail:string,
    RiskLastUpdatedDateTime:datetime,
    RiskLevel:string,
    RiskState:string,
    UserDisplayName:string,
    UserPrincipalName:string
)
[
    datetime(2026-03-05T14:22:00Z), "riskcorr-identity-047", "a7d2f8c1-4e2b-48f3-9a06-2e5c7b3d9f1a", false, false, "RiskyUserDetected", "adminConfirmedUserCompromised", datetime(2026-02-28T10:15:00Z), "high", "confirmedCompromised", "U1291", "u1291@contoso.onmicrosoft.com",
    datetime(2026-03-05T15:18:00Z), "riskcorr-identity-052", "b9e4a6d2-7f3c-41b8-a2e5-3f6c8d1e2b4a", false, false, "RiskyUserUpdated",  "anonymousIpAddress",             datetime(2026-02-27T14:42:00Z), "medium", "atRisk",              "U3415", "u3415@contoso.onmicrosoft.com",
    datetime(2026-03-05T16:33:00Z), "riskcorr-identity-059", "c5f1b7e3-9d4a-42c1-b8f6-5e7d9c2a3f5b", false, false, "RiskyUserUpdated",  "userPerformedSecuredPasswordReset", datetime(2026-02-26T11:20:00Z), "medium", "remediated",          "U2847",   "u2847@contoso.onmicrosoft.com",
];

Next steps

Your data lake is now populated with sample telemetry from IdentityDrift, Microsoft Entra ID, and Microsoft Defender. Use this data to test and develop your Security Copilot agent. See Use an MCP tool in Visual Studio Code to explore Sentinel MCP Server tools and query the data using prompts.

Before proceeding to building Security Copilot Agent for IdentityDrift use case, explore the MCP server with the following prompts to query the sample data ingested in this step to learn how to query the data from Sentinel data lake using Sentinel MCP.

Sample prompts for Sentinel MCP via VS Code Chat:

  • Search for tables related to "identity" or "signin" in my workspace.
  • Show me process execution events for user u1291 in DeviceProcessEvents_KQL_CL — what commands did they run?
  • Correlate activity for u1291@contoso.onmicrosoft.com: check their sign-in risk in SigninLogs_KQL_CL, their access in CommonSecurity_ID_KQL_CL, and any process execution in DeviceProcessEvents_KQL_CL. Summarize the attack chain.

Continue to Build a Security Copilot agent

  • Last updated on