Monitor the health of your data connectors

After you've configured and connected your Microsoft Sentinel workspace to your data connectors, you'll want to monitor your connector health, viewing any service or data source issues, such as authentication, throttling, and more.

You also might like to configure notifications for health drifts for relevant stakeholders who can take action. For example, configure email messages, Microsoft Teams messages, new tickets in your ticketing system, and so on.

This article describes how to use the following features, which allow you to keep track of your data connectors' health, connectivity, and performance from within Microsoft Sentinel:

  • Data connectors health monitoring workbook. This workbook provides additional monitors, detects anomalies, and gives insight regarding the workspace’s data ingestion status. You can use the workbook’s logic to monitor the general health of the ingested data, and to build custom views and rule-based alerts.

  • SentinelHealth data table. (Public preview) Provides insights on health drifts, such as latest failure events per connector, or connectors with changes from success to failure states, which you can use to create alerts and other automated actions. The SentinelHealth data table is currently supported only for selected data connectors.

Important

The SentinelHealth data table is currently in PREVIEW. See the Supplemental Terms of Use for Microsoft Azure Previews for additional legal terms that apply to Azure features that are in beta, preview, or otherwise not yet released into general availability.

Use the health monitoring workbook

  1. From the Microsoft Sentinel portal, select Workbooks from the Threat management menu.

  2. In the Workbooks gallery, enter health in the search bar, and select Data collection health monitoring from among the results.

  3. Select View template to use the workbook as is, or select Save to create an editable copy of the workbook. When the copy is created, select View saved workbook.

  4. Once in the workbook, first select the subscription and workspace you wish to view, then define the TimeRange to filter the data according to your needs. Use the Show help toggle to display in-place explanation of the workbook.

    data connector health monitoring workbook landing page

There are three tabbed sections in this workbook:

  • The Overview tab shows the general status of data ingestion in the selected workspace: volume measures, EPS rates, and time last log received.

  • The Data collection anomalies tab will help you to detect anomalies in the data collection process, by table and data source. Each tab presents anomalies for a particular table (the General tab includes a collection of tables). The anomalies are calculated using the series_decompose_anomalies() function that returns an anomaly score. Learn more about this function. Set the following parameters for the function to evaluate:

    • AnomaliesTimeRange: This time picker applies only to the data collection anomalies view.

    • SampleInterval: The time interval in which data is sampled in the given time range. The anomaly score is calculated only on the last interval's data.

    • PositiveAlertThreshold: This value defines the positive anomaly score threshold. It accepts decimal values.

    • NegativeAlertThreshold: This value defines the negative anomaly score threshold. It accepts decimal values.

      data connector health monitoring workbook anomalies page

  • The Agent info tab shows you information about the health of the Log Analytics agents installed on your various machines, whether Azure VM, other cloud VM, on-premises VM, or physical. You can monitor the following:

    • System location

    • Heartbeat status and latency

    • Available memory and disk space

    • Agent operations

      In this section you must select the tab that describes your machines’ environment: choose the Azure-managed machines tab if you want to view only the Azure Arc-managed machines; choose the All machines tab to view both managed and non-Azure machines with the Log Analytics agent installed.

      data connector health monitoring workbook agent info page

Use the SentinelHealth data table (Public preview)

To get data connector health data from the SentinelHealth data table, you must first turn on the Microsoft Sentinel health feature for your workspace. For more information, see Turn on health monitoring for Microsoft Sentinel.

Once the health feature is turned on, the SentinelHealth data table is created at the first success or failure event generated for your data connectors.

Supported data connectors

The SentinelHealth data table is currently supported only for the following data connectors:

Understanding SentinelHealth table events

The following types of health events are logged in the SentinelHealth table:

  • Data fetch status change. Logged once an hour as long as a data connector status remains stable, with either continuous success or failure events. For as long as a data connector's status does not change, monitoring only hourly works to prevent redundant auditing and reduce table size. If the data connector's status has continuous failures, additional details about the failures are included in the ExtendedProperties column.

    If the data connector's status changes, either from a success to failure, from failure to success, or has changes in failure reasons, the event is logged immediately to allow your team to take proactive and immediate action.

    Potentially transient errors, such as source service throttling, are logged only after they've continued for more than 60 minutes. These 60 minutes allow Microsoft Sentinel to overcome a transient issue in the backend and catch up with the data, without requiring any user action. Errors that are definitely not transient are logged immediately.

  • Failure summary. Logged once an hour, per connector, per workspace, with an aggregated failure summary. Failure summary events are created only when the connector has experienced polling errors during the given hour. They contain any extra details provided in the ExtendedProperties column, such as the time period for which the connector's source platform was queried, and a distinct list of failures encountered during the time period.

For more information, see SentinelHealth table columns schema.

Run queries to detect health drifts

Create queries on the SentinelHealth table to help you detect health drifts in your data connectors. For example:

Detect latest failure events per connector:

SentinelHealth
| where TimeGenerated > ago(3d)
| where OperationName == 'Data fetch status change'
| where Status in ('Success', 'Failure')
| summarize TimeGenerated = arg_max(TimeGenerated,*) by SentinelResourceName, SentinelResourceId
| where Status == 'Failure'

Detect connectors with changes from fail to success state:

let lastestStatus = SentinelHealth
| where TimeGenerated > ago(12h)
| where OperationName == 'Data fetch status change'
| where Status in ('Success', 'Failure')
| project TimeGenerated, SentinelResourceName, SentinelResourceId, LastStatus = Status
| summarize TimeGenerated = arg_max(TimeGenerated,*) by SentinelResourceName, SentinelResourceId;
let nextToLastestStatus = SentinelHealth
| where TimeGenerated > ago(12h)
| where OperationName == 'Data fetch status change'
| where Status in ('Success', 'Failure')
| join kind = leftanti (lastestStatus) on SentinelResourceName, SentinelResourceId, TimeGenerated
| project TimeGenerated, SentinelResourceName, SentinelResourceId, NextToLastStatus = Status
| summarize TimeGenerated = arg_max(TimeGenerated,*) by SentinelResourceName, SentinelResourceId;
lastestStatus
| join kind=inner (nextToLastestStatus) on SentinelResourceName, SentinelResourceId
| where NextToLastStatus == 'Failure' and LastStatus == 'Success'

Detect connectors with changes from success to fail state:

let lastestStatus = SentinelHealth
| where TimeGenerated > ago(12h)
| where OperationName == 'Data fetch status change'
| where Status in ('Success', 'Failure')
| project TimeGenerated, SentinelResourceName, SentinelResourceId, LastStatus = Status
| summarize TimeGenerated = arg_max(TimeGenerated,*) by SentinelResourceName, SentinelResourceId;
let nextToLastestStatus = SentinelHealth
| where TimeGenerated > ago(12h)
| where OperationName == 'Data fetch status change'
| where Status in ('Success', 'Failure')
| join kind = leftanti (lastestStatus) on SentinelResourceName, SentinelResourceId, TimeGenerated
| project TimeGenerated, SentinelResourceName, SentinelResourceId, NextToLastStatus = Status
| summarize TimeGenerated = arg_max(TimeGenerated,*) by SentinelResourceName, SentinelResourceId;
lastestStatus
| join kind=inner (nextToLastestStatus) on SentinelResourceName, SentinelResourceId
| where NextToLastStatus == 'Success' and LastStatus == 'Failure'

Configure alerts and automated actions for health issues

While you can use the Microsoft Sentinel analytics rules to configure automation in Microsoft Sentinel logs, if you want to be notified and take immediate action for health drifts in your data connectors, we recommend that you use Azure Monitor alert rules.

For example:

  1. In an Azure Monitor alert rule, select your Microsoft Sentinel workspace as the rule scope, and Custom log search as the first condition.

  2. Customize the alert logic as needed, such as frequency or lookback duration, and then use queries to search for health drifts.

  3. For the rule actions, select an existing action group or create a new one as needed to configure push notifications or other automated actions such as triggering a Logic App, Webhook, or Azure Function in your system.

For more information, see Azure Monitor alerts overview and Azure Monitor alerts log.

SentinelHealth table columns schema

The following table describes the columns and data generated in the SentinelHealth data table for data connectors:

ColumnName ColumnType Description
TenantId String The tenant ID for your Microsoft Sentinel workspace.
TimeGenerated Datetime The time at which the health event occurred.
OperationName String The health operation. One of the following values:

-Data fetch status change for health or success indications
- Failure summary for aggregated health summaries.

For more information, see Understanding SentinelHealth table events.
SentinelResourceId String The unique identifier of the Microsoft Sentinel workspace and the associated connector on which the health event occurred.
SentinelResourceName String The data connector name.
Status String Indicates Success or Failure for the Data fetch status change OperationName, and Informational for the Failure summary OperationName.
Description String Describes the operation, including extended data as needed. For example, for failures, this column might indicate the failure reason.
WorkspaceId String The workspace GUID on which the health issue occurred. The full Azure Resource Identifier is available in the SentinelResourceID column.
SentinelResourceType String The Microsoft Sentinel resource type being monitored: Data connector
SentinelResourceKind String The type of data connector being monitored, such as Office365.
RecordId String A unique identifier for the record that can be shared with the support team for better correlation as needed.
ExtendedProperties Dynamic (json) A JSON bag that varies by the OperationName value and the Status of the event:

- For Data fetch status change events with a success indicator, the bag contains a ‘DestinationTable’ property to indicate where data from this connector is expected to land. For failures, the contents vary depending on the failure type.
Type String SentinelHealth

Next steps

Learn how to onboard your data to Microsoft Sentinel, connect data sources, and get visibility into your data, and potential threats.