Mismatch in amount of data received in logs analytics workspace and DCR metrics

Ashwin Venkatesha 105

I have defined a data collection rule and am using logs ingestion api to send data to 2 custom tables.

I have defined diagnostic settings for the DCR such that error logs are sent to logs analytics workspace.

For about an hour, I have events ingested into custom tables using logs ingestion api.

From the query in logs analytics workspaces,

let Count1 = toscalar(table('FlowLogs_CL') | summarize count());
let Count2 = toscalar(table('CustomLog_CL') | summarize count());
print TotalCount = Count1 + Count2

Answer: 324

However, from the metrics "logs rows received per min", the number is quite different

User's image

Note, that there are 0 rows dropped during ingestion, no errors reported by DCRErrorLogs

Why is it that the query results are not matching?

Booker Vail 0 Reputation points

2024-03-28T08:10:01.6533333+00:00
Hello, @Ashwin Venkatesha

I can see your post And give you my best to solve your query. Please see information below and follow step by step.

he mismatch between the query results in your Log Analytics workspace and the metrics for “logs rows received per min” could be due to several factors.

See here are some steps you can take to troubleshoot and resolve the issue:

Ensure that the time period you’re querying in the Log Analytics workspace matches the time frame of the metrics you’re viewing. Time zone differences or delays in data processing can cause discrepancies.

There might be a delay in data ingestion, which means the data has reached Azure Monitor but hasn’t been fully ingested into the custom tables yet. This can temporarily cause the numbers to differ see here: https://learn.microsoft.com/en-us/azure/azure-monitor/essentials/gm-socrates /data-collection-monitor

Verify that the scope of your query is correct and that it includes all relevant tables and data types. Ensure that the tables FlowLogs_CL and CustomLog_CL are the only sources of the data you’re expecting.

Check the diagnostic settings for the DCR to confirm that all necessary data types are being collected and that there are no filters that might exclude certain data.

Review the accuracy of the metrics “logs rows received per min”. Sometimes, the metrics might not update in real-time or could be averaged over a period, which might not reflect the exact count at a specific minute.

Review the configuration of your Data Collection Rule to ensure it’s set up correctly to collect and send the data you’re expecting to the Log Analytics workspace.

Even though no errors are reported by DCRErrorLogs, it’s worth checking the DCRLogErrors table in the Log Analytics workspace for any unnoticed issues.

Refer to the Azure Monitor documentation for detailed guidance on monitoring and troubleshooting DCR data collection.

If these steps do not resolve the discrepancy, please tell me I am happy to help you.

I hope my suggestion is helpful for you.

Best Regard,
Booker Vail
AnuragSingh-MSFT 19,856 Reputation points

2024-03-28T09:57:20.45+00:00
Ashwin Venkatesha, Could you please answer the questions below:

What is the expected count of logs sent to log ingestion API? The y-axis reports number in thousands - are you sending this many logs to the API?

The sum of log entries available in custom tables is 324 (as you have shown in the output above). Can you please check in these custom tables, that the log entries are between 12:15AM and 12:45AM only?

Based on my understanding, the screenshot above is taken from a workbook where metric chart is pinned. Instead of accessing the workbook, can you check the "Metrics" option in DCR itself to see the entries - are these entries also in thousands? I am trying to understand if the issue is specific to the Workbook and not the Metric explorer of the DCR

Please let us know if you have any questions
Ashwin Venkatesha 105 Reputation points

2024-04-01T06:30:30.21+00:00

Hi @AnuragSingh-MSFT ,

Here is an updated state after more traffic ingestion during a specific timeframe May 30th 12am to 10am.

Notice, that this metric says 101.82M.
Lets look at what logs ingestion says,

Note that the time chosen in time range is the same as what is in the metrics view.

The only transformation that happens when data is ingested is, to rename a column (timestamp to Timegenerated).

I do not understand how 101.82M has to be compared with 40M?
Clive Watson 5,711 Reputation points MVP

2024-04-03T10:25:09.3866667+00:00

Just a thought, does 'rows received' also include 'dropped rows' e.g. are you dropping 60m, hence the Tables only receiving 40m?

https://learn.microsoft.com/en-us/azure/azure-monitor/essentials/data-collection-monitor#dcr-metrics
Ashwin Venkatesha 105 Reputation points

2024-04-03T17:48:18.4366667+00:00

@Clive Watson I did check the dropped rows and there were none.
Ashwin Venkatesha 105 Reputation points

2024-04-03T17:49:21.41+00:00

@Clive Watson There are 0 dropped rows, I checked that too for the time duration.

Accepted answer

AnuragSingh-MSFT 19,856 Reputation points

2024-04-05T05:01:28.55+00:00

Ashwin Venkatesha, thank you for the clarification. I understand the scenario now.

The numbers as reported for "Log Rows Received per Min (Sum)" generally will not be equal to "Actual count of rows" available in the custom table. The following section discusses it in detail:

Log Rows Received per Min (Sum) - This metric simply sums up the counter values received (which is "Log Rows Received per Min"). Notice the "per min" aggregation happening here. The sum of this aggregation is not going to be equal to the "actual count" of logs ingested. If the counter were "count of rows received", the sum would have been equal to count of rows - which is not the case here.

Also, notice that this value changes depending on the granularity chosen from the filter above:

In short, these are 2 different values and should not be compared or expected to be equal.

Hope this helps.
Please sign in to rate this answer.
Ashwin Venkatesha 105 Reputation points

2024-04-05T19:20:24.96+00:00

@AnuragSingh-MSFT

Thanks for the clarification, does that mean

At 12:32 AM, "739720" events were seen, which equates to 12k events/sec ?

AnuragSingh-MSFT 19,856 Reputation points

2024-04-08T05:44:24.96+00:00

@Ashwin Venkatesha thank you for the reply.

To answer your question - not exactly. This is partially correct.

Refer the "Time range and Granularity" filter at the top right corner of this view:

If the "Time granularity" is set to "Automatic" (which is 5 minutes), it would mean than in the last 5 minutes (739.72k X 5) Low rows were received. Because the aggregation is "per min", the total rows received in the last 5 minutes was divided by 5 to get this number.

Note that the logs are not collected near real time as they are written in the log file. There are 2 main concepts that come in play to avoid excessive network load and achieve optimum performance:

batching - Log rows are buffered locally on the machine in batches, before they are sent out to Azure Monitor. This batching is dynamic in nature and depends on multiple attributes like row count, size of buffer etc.

Frequency - Logs are also uploaded every X seconds/minute (irrespective of whether the batching has exceeded the batch threshold)

Because of the above 2 concepts, the log rows being received per minute in the Azure Monitor might slightly be different from how they are being written in the log files.

I would suggest reviewing the log file and collected logs to check that all the rows are indeed present and getting collected.

Hope this helps.

Ashwin Venkatesha 105 Reputation points

2024-04-08T15:54:34.4133333+00:00

@AnuragSingh-MSFT

Thanks, that makes sense, however I am missing the view of 'Time granularity'.

Time range lets me select a certain range which is in your screenshot as well.

What I understand from your explanation is, if I assume time granularity is 5mins, then 739.72k logs were received every minute.

Q1: How to find time granularity?

AnuragSingh-MSFT 19,856 Reputation points

2024-04-10T04:48:09.6466667+00:00

Ashwin Venkatesha thank you for the reply.

You are correct about the 739.72k count of log rows per minute. But this is only true for the last 5 minutes (assuming that the granularity is set at 5 minutes).

Primarily, there are 2 ways to view this metric for "Log Rows Received per minutes"

From the "Metrics" menu of the "Data collection Rule" itself, as shown below. The screenshot also highlights the filter where you can set the range and time granularity:

The second option is using what you have shared in the screenshot. In this case, the "Metrics" view (as shown in point 1) is added to a workbook. The granularity option is available here when you click on "Edit" option as shown below:

I checked and time granularity can not be added as a parameter at the moment in workbook. Hence you can have the filter for "Time range" but not for "granularity" for your users. The only way to set granularity is to edit the workbook section to have it set for every user of workbook.

Hope this helps.

If the discussion on this thread has helped, please click Accept answer so that it can help others in the community looking for help on similar topics.

AnuragSingh-MSFT 19,856 Reputation points

2024-04-12T05:42:52.08+00:00

Ashwin Venkatesha Following-up to check if you had a chance to review the answer provided.

If the answer did not help, please add more context/follow-up question for it. Else, if the answer helped, please click Accept answer so that it can help others in the community looking for help on similar topics.

Deleted

This comment has been deleted due to a violation of our Code of Conduct. The comment was manually reported or identified through automated detection before action was taken. Please refer to our Code of Conduct for more information.
Sign in to comment

Mismatch in amount of data received in logs analytics workspace and DCR metrics

0 additional answers