Need confirmation on exporting VNet flow logs to Event hub

Question

Issue: Azure VNet Flow Logs: Discrepancy between Storage Account and Event Hub logs after configuring Traffic Analytics export rule

We have created  export rule to send the logs of Virtual network to Event hub by using below steps and followed the below document to setup this :

1. Created Virtual Networks : https://learn.microsoft.com/en-us/azure/virtual-network/quickstart-create-virtual-network?tabs=portal#create-a-virtual-network-and-an-azure-bastion-host
2. Created a Log Analytics Workspace : https://learn.microsoft.com/en-us/azure/azure-monitor/logs/quick-create-workspace?tabs=azure-portal
3. Enable Network Watcher : https://learn.microsoft.com/en-us/azure/network-watcher/network-watcher-create?tabs=portal#enable-network-watcher-for-your-region
4. Enable VNet flow log : https://learn.microsoft.com/en-us/azure/network-watcher/vnet-flow-logs-manage?tabs=portal
5. Created Traffic Analytics

. Configured Log Analytics Workspace

. Created Export Rule with:

• Table: NTANetAnalytics
• Destination: Event Hub
• Required subscription and workspace details

https://learn.microsoft.com/en-us/azure/network-watcher/vnet-flow-logs-manage?tabs=portal 

We are getting the complete logs in storage account but when we are comparing the logs between from storage account & event hub then we are not getting complete logs in Event hub. Could you please help here ?

Answer 1

Hello Ayush Shrimal

I understand that you're dealing with an issue where your Azure VNet flow logs are successfully exporting to a storage account, but you're noticing discrepancies when comparing those logs to what's being sent to the Event Hub.

You are on the right track with the configuration and set up based on the resources you've provided. Here are a few possible reasons and troubleshooting steps:

• Traffic analytics exports segregated data, not every individual flow. You will not get 1:1 flow entries compared to storage. Some flows may not appear if they are too short-lived or below aggregation threshold.
• Log Processing Delay: Traffic Analytics processes logs at intervals, so there might be a delay in logs appearing in Event Hub compared to Storage Account. Check if logs appear after some time.
• Filtering in Export Rule: Ensure that the export rule isn't filtering out specific logs before sending them to Event Hub.
• Event Hub Quotas & Limits: Event Hub has throughput units and retention limits. If logs exceed these limits, some data might not be stored. Check Event Hub metrics to see if any messages were dropped.
• Log Format Differences: Storage Account logs are stored in JSON format, while Event Hub logs might be processed differently.

Please don’t forget to close the thread by clicking "Accept the answer" and "Yes" wherever the information provided helps you, as this can be beneficial to other community members.

Answer 2

Dear Ayush,

Thank you for reaching out regarding the discrepancy between VNet Flow Logs in your Storage Account and Event Hub. Based on your description, you’ve configured the export correctly, but the logs in Event Hub appear incomplete compared to Storage. Here are some troubleshooting steps:

First, verify that the export rule for the NTANetAnalytics table is correctly set up with the right Event Hub destination and that the Event Hub has enough throughput units to handle the log volume. Also, check if there are any retention or capture settings in the Event Hub namespace that might be filtering or truncating logs.

Next, consider the processing time of Traffic Analytics, which aggregates logs every 10–60 minutes. If logs are missing in Event Hub but present in Storage, it could be a delay wait 1–2 hours to see if they sync.

Additionally, review the diagnostic settings for NSG Flow Logs under Network Watcher to ensure Event Hub streaming is enabled with the same retention settings as Storage. Check Azure Monitor metrics for the Event Hub (e.g., Incoming Messages) to confirm data is being ingested. If metrics show activity but logs are incomplete, the issue might be on the consumer side.

If the problem persists, please share screenshots of your export rule and Traffic Analytics configuration, as well as any errors from the Event Hub’s dead-letter queue or audit logs. For deeper investigation, consider opening a support request in the Azure Portal with relevant correlation IDs.

Let me know if you need further assistance!

Best regards,
Alex
P.S. If my answer help to you, please Accept my answer
PPS That is my Answer and not a Comment