Data stopped in log analytics

Question

Data ingestion stopped at time. What is the practice need to follow daily cap ingestion rate limit for diagnostics? Also what altering setup should be used to notify on these conditions.

Answer 1

Hello Williams

Thank you for posting your query on Microsoft Q&A platform.

The daily cap limits the amount of data that a Log Analytics workspace can collect in a day. When the daily cap is reached, data collection stops until the reset time. You can either wait for collection to automatically restart, or increase the daily data volume limit.

Check Log Analytics workspace data collection status: When data collection in a Log Analytics workspace stops, an event with a type of Operation is created in the workspace. Run the following query to check whether you're reaching the daily limit and missing data:

Kusto

Operation | where OperationCategory == 'Data Collection Status'

When data collection stops, the OperationStatus is Warning. When data collection starts, the OperationStatus is Succeeded.

To be notified when data collection stops, use the steps described in the Alert when daily cap is reached section. To configure an e-mail, webhook, or runbook action for the alert rule, use the steps described in create an action group.

Check whether your workspace reached its data ingestion rate limit

Use this query to retrieve the record that indicates the data ingestion rate limit was reached.

Kusto

Operation 
| where OperationCategory == "Ingestion" 
| where Detail startswith "The rate of data crossed the threshold"

Azure Monitor Agent not sending data: Azure Monitor Agent collects data from virtual machines and sends the data to Azure Monitor. An agent might stop sending data to your Log Analytics workspace in various scenarios. For example, when Azure Site Recovery recovers a virtual machine in a disaster recovery scenario, the resource ID of the machine changes, requiring reinstallation of Azure Monitor Agent on the machine.

Check the health of agents sending data to your workspace: Azure Monitor Agent instances installed on all virtual machines that send data to your Log Analytics workspace send a heartbeat to the Heartbeat table every minute.

Run this query to list VMs that haven't reported a heartbeat in the last five minutes:

Kusto

Heartbeat 
| where TimeGenerated > ago(24h)
| summarize LastCall = max(TimeGenerated) by Computer, _ResourceId
| where LastCall < ago(5m)

Thanks,

Suchitra.

Answer 2

When data ingestion stops in Azure Log Analytics due to reaching the daily cap, it's important to follow certain practices to manage the ingestion rate effectively and set up alerts for notifications.

Practices for Managing Daily Cap Ingestion Rate:

1. Understand the Daily Cap: The daily cap is a safeguard against unexpected spikes in data volume that could lead to unplanned charges. It should not be used as a primary cost-control tool but rather as a backup measure for rare or unforeseen data surges.
2. Use Ingestion-Time Transformations: Instead of relying solely on the daily cap, use ingestion-time transformations to filter or reshape data before it is ingested into the Log Analytics workspace. This helps in managing the volume of data collected more effectively.
3. Monitor Data Collection Status: Regularly check the data collection status in your Log Analytics workspace to ensure you are not consistently hitting the daily cap. Use queries to monitor the ingestion operations and identify any patterns that may lead to reaching the cap.

Alerting Setup for Notifications:

1. Create Alerts for Daily Cap Reached: Set up alerts to notify you when the data collection stops due to the daily cap being reached. This can be done by creating an alert rule based on the "Data collection stopped" operation event.
2. Use the _LogOperation Table: Monitor the _LogOperation table for events related to data collection status. You can run queries to check for collection stopped and resumed events, which can help you understand when data collection is affected.
3. Configure Action Groups: For the alerts, configure action groups to send notifications via email, webhook, or other methods when the daily cap is reached. This ensures that you are promptly informed of any issues with data ingestion.

By following these practices and setting up appropriate alerts, you can better manage the ingestion limits and stay informed about the status of your data collection in Azure Log Analytics.

---

References:

• Set daily cap limit for Log Analytics workspace
• Set daily cap on Log Analytics workspace
• Monitor operational issues in your Azure Monitor Log Analytics workspace
• Troubleshoot why data is no longer being collected in Azure Monitor

AI-generated content may be incorrect. Read our transparency notes for more information.