Missing Records in Azure Log Analytics

Question

Hi All,

Logs of activities, pipelines, and triggers are gathered within the Synapse environment for monitoring purposes. It has been observed that some pipelines ran successfully, as seen in the Monitor section of Synapse, but there are missing records in the logs. For example, one pipeline only has a record with "Succeeded" status, while "Queued" and "InProgress" are missing. What's more, another pipeline has both "Queued" and "InProgress" statuses but is missing the "Succeeded" record.

What could explain the absence of these records in the logs despite the successful completion in Synapse?

Any insights or potential solutions would be appreciated!

Thanks,

Jakub

Answer 1

Hi Jakub Pytlik,

Thanks for reaching out to Microsoft Q&A.

The missing log records in Azure Log Analytics for Synapse pipelines, despite their successful completion, could be attributed to several potential causes:

1. Latency in Log Analytics Data Ingestion: Azure Log Analytics might experience delays in ingesting log data, causing certain records (ex: "Queued", "InProgress", or "Succeeded" statuses) to be missing temporarily. In such cases, the records should eventually appear after the ingestion delay resolves.

Log Data Retention Policy or Log Filtering: It’s possible that a retention policy or custom log filtering could be omitting certain pipeline statuses. Check your Log Analytics workspace settings to ensure that all necessary data types are being collected without exclusions, and verify that there are no filters set that might exclude "Queued", "InProgress", or "Succeeded" records.

Issues with Diagnostic Settings Configuration: If the diagnostic settings for the Synapse workspace aren’t correctly configured, some logs may not be captured. Ensure that all pipeline logs, including the required categories (such as PipelineRuns, TriggerRuns, ActivityRuns), are being sent to Log Analytics. Double-check the scope of the diagnostics and the selected log categories.

Intermittent Connectivity or Pipeline Execution Issues: If the pipeline execution experiences intermittent connectivity issues (e.g., network issues), some log entries might not be transmitted to Log Analytics even though the pipeline ultimately completes successfully. This may explain missing "Queued" or "InProgress" records.

Concurrent Pipeline Execution Conflicts: If multiple pipelines are running concurrently, log entry conflicts may occur, causing incomplete logs for individual pipelines. Ensure that the concurrency settings in Synapse and Log Analytics are appropriately configured to handle parallel runs.

Synapse Monitor Inconsistencies: There could be inconsistencies between the Synapse Monitor and Log Analytics records. The Monitor section in Synapse provides a near-real-time status but might not always perfectly sync with Log Analytics. Reviewing both can sometimes highlight discrepancies that aren't actual data loss but reflect a lag between the systems.

Potential Solutions:

• Investigate Diagnostic Settings: Review your Synapse diagnostic settings to ensure that all relevant pipeline statuses (Queued, InProgress, Succeeded) are being logged and sent to Log Analytics.
• Check Log Analytics Filters: Ensure no filters are in place that might inadvertently filter out important logs.
• Review Pipeline Execution Logs: Manually cross-check the logs in both Synapse Monitor and Log Analytics to identify any patterns in the missing records, which could help pinpoint the issue.
• Monitor Ingestion Delays: Check Azure Service Health to see if there are any ongoing issues related to Log Analytics that might explain delayed or missing records.

By exploring these areas, you should be able to better understand why some pipeline statuses are missing in your logs and find a solution to ensure all records are captured.

Please feel free to click the 'Upvote' (Thumbs-up) button and 'Accept as Answer'. This helps the community by allowing others with similar queries to easily find the solution.

Answer 2

Thanks for picking this up!

1. Latency in Log Analytics Data Ingestion: those missing recorde have been missing since 8th October 2024, so the would've already appeared.
2. Log Data Retention Policy or Log Filtering: we've been gathering those records for over half a year already and every finished pipeline comes with 3 statuses
3. Issues with Diagnostic Settings Configuration: settings are correct for PipelineRuns, TriggerRuns, ActivityRuns as we gather those logs on daily basis
4. Intermittent Connectivity or Pipeline Execution Issues: How can I verify if there was a connectivity issues?
5. Concurrent Pipeline Execution Conflicts: multiple pipelines run concurrently every time, this wouldn't something new to our system
6. Synapse Monitor Inconsistencies: I wouldn't expect the lag to last 2 weeks

I checked Azure Service Health and couldn't find anything related to Logs Analytics. Are there any other possibilities for those missing logs?

Answer 3

Hii Jakub Pytlik

Welcome to the Microsoft Q&A Platform! Thank you for asking your question here.

Sometimes, specific resources within Synapse may encounter issues. Check the individual activity runs within the pipeline for any errors or warnings that might not have surfaced at the pipeline level.

If your Synapse workspace is connected to Azure Log Analytics, make sure the settings are correct and that all important logs are being sent.

https://learn.microsoft.com/en-us/azure/synapse-analytics/security/connectivity-settings

Check how long your logs are kept. If they’re deleted based on certain rules, you might not see some records. Make sure the retention policies aren’t hiding any logs.

https://learn.microsoft.com/en-us/purview/audit-log-retention-policies?tabs=microsoft-purview-portal

If pipelines run at the same time, they can cause logging issues. Make sure to manage them properly to avoid problems.

Different trigger types (manual, scheduled, tumbling window) can affect how logging works. Check for any differences based on how the pipelines were started.
https://learn.microsoft.com/en-us/azure/data-factory/concepts-pipeline-execution-triggers

If you have custom logging in your pipelines, make sure it’s working right. Problems with it can cause logs to be missing.

https://learn.microsoft.com/en-us/azure/devops/pipelines/troubleshooting/review-logs?view=azure-devops&tabs=windows-agent

If you updated your pipeline recently, make sure you’re checking the right version that’s currently active. Older versions might still be running, which can confuse the logs.

If you are checking logs from different time zones, use the right time settings to avoid confusion about the statuses.
To check for connectivity issues, review network metrics and use Azure Monitor to find any alerts or problems during the time the logs were missing.
https://learn.microsoft.com/en-us/azure/network-watcher/network-insights-overview

If you’ve tried all these options, getting in touch with Azure Support might help find any backend issues.
If you have any further queries, do let us know.

---

If the answer is helpful, please click "Accept Answer" and "Upvote it".