Edit

Share via

Traffic analytics schema and data aggregation

Traffic analytics is a cloud-based solution that provides visibility into user and application activity in cloud networks. Traffic analytics analyzes Azure Network Watcher flow logs to provide insights into traffic flow in your Azure cloud. With traffic analytics, you can:

  • Visualize network activity across your Azure subscriptions and identify hot spots.
  • Identify security threats, and secure your network, with information such as open-ports, applications attempting internet access, and virtual machines (VMs) connecting to rogue networks.
  • Understand traffic flow patterns across Azure regions and the internet to optimize your network deployment for performance and capacity.
  • Pinpoint network misconfigurations leading to failed connections in your network.
  • Know network usage in bytes, packets, or flows.

Data aggregation

  • All flow logs between FlowIntervalStartTime and FlowIntervalEndTime are captured at one-minute intervals as blobs in a storage account.
  • Default processing interval of traffic analytics is 60 minutes, meaning that every hour, traffic analytics picks blobs from the storage account for aggregation. However, if a processing interval of 10 minutes is selected, traffic analytics will instead pick blobs from the storage account every 10 minutes.
  • Flows that have the same Source IP, Destination IP, Destination port, NSG name, NSG rule, Flow Direction, and Transport layer protocol (TCP or UDP) are clubbed into a single flow by traffic analytics (Note: source port is excluded for aggregation).
  • This single record is decorated (details in the section below) and ingested in Azure Monitor logs by traffic analytics. This process can take up to 1 hour.
  • FlowStartTime field indicates the first occurrence of such an aggregated flow (same four-tuple) in the flow log processing interval between FlowIntervalStartTime and FlowIntervalEndTime.
  • For any resource in traffic analytics, the flows indicated in the Azure portal are total flows seen, but in Azure Monitor logs, user sees only the single, reduced record. To see all the flows, use the blob_id field, which can be referenced from storage. The total flow count for that record matches the individual flows seen in the blob.

The following query helps you look at all subnets interacting with non-Azure public IPs in the last 30 days.

NTANetAnalytics
| where SubType == "FlowLog" and FlowStartTime >= ago(30d) and FlowType == "ExternalPublic"
| project SrcSubnet, DestSubnet

Traffic analytics schema

Traffic analytics is built on top of Azure Monitor logs, so you can run custom queries on data decorated by traffic analytics and set alerts.

The following table lists the fields in the schema and what they signify for virtual network flow logs. For more information, see NTANetAnalytics.

Field Format Comments
TableName NTANetAnalytics Table for traffic analytics data.
SubType FlowLog Subtype for the flow logs. Use only FlowLog, other values of SubType are for internal use.
FASchemaVersion 3 Schema version. Doesn't reflect virtual network flow log version.
TimeProcessed Date and time in UTC Time at which the traffic analytics processed the raw flow logs from the storage account.
FlowIntervalStartTime Date and time in UTC Starting time of the flow log processing interval (time from which flow interval is measured).
FlowIntervalEndTime Date and time in UTC Ending time of the flow log processing interval.
FlowStartTime Date and time in UTC First occurrence of the flow (which gets aggregated) in the flow log processing interval between FlowIntervalStartTime and FlowIntervalEndTime. This flow gets aggregated based on aggregation logic.
FlowEndTime Date and time in UTC Last occurrence of the flow (which gets aggregated) in the flow log processing interval between FlowIntervalStartTime and FlowIntervalEndTime.
FlowType - IntraVNet
- InterVNet
- S2S
- P2S
- AzurePublic
- ExternalPublic
- MaliciousFlow
- Unknown Private
- Unknown		 See Notes for definitions.
SrcIp Source IP address Blank in AzurePublic and ExternalPublic flows.
DestIp Destination IP address Blank in AzurePublic and ExternalPublic flows.
TargetResourceId ResourceGroupName/ResourceName The ID of the resource at which flow logging and traffic analytics is enabled.
TargetResourceType VirtualNetwork/Subnet/NetworkInterface Type of resource at which flow logging and traffic analytics is enabled (virtual network, subnet, NIC or network security group).
FlowLogResourceId ResourceGroupName/NetworkWatcherName/FlowLogName The resource ID of the flow log.
DestPort Destination Port Port at which traffic is incoming.
L4Protocol - T
- U		 Transport Protocol. T = TCP
U = UDP
L7Protocol Protocol Name Derived from destination port.
FlowDirection - I = Inbound
- O = Outbound		 Direction of the flow: in or out of the target resource per flow log.
FlowStatus - A = Allowed
- D = Denied		 Status of flow: allowed or denied by target resource per flow log.
AclList <SubscriptionID>/<resourcegroup_Name>/<NSG_Name> Network security group associated with the flow.
AclRule NSG_Rule_Name Network security group rule that allowed or denied the flow.
MACAddress MAC Address MAC address of the NIC at which the flow was captured.
SrcSubscription Subscription ID Subscription ID of virtual network / network interface / virtual machine that the source IP in the flow belongs to.
DestSubscription Subscription ID Subscription ID of virtual network / network interface / virtual machine that the destination IP in the flow belongs to.
SrcRegion Azure Region Azure region of virtual network / network interface / virtual machine that the source IP in the flow belongs to.
DestRegion Azure Region Azure region of virtual network that the destination IP in the flow belongs to.
SrcNic <resourcegroup_Name>/<NetworkInterfaceName> NIC associated with the source IP in the flow.
DestNic <resourcegroup_Name>/<NetworkInterfaceName> NIC associated with the destination IP in the flow.
SrcVm <resourcegroup_Name>/<VirtualMachineName> Virtual machine associated with the source IP in the flow.
DestVm <resourcegroup_Name>/<VirtualMachineName> Virtual machine associated with the destination IP in the flow.
SrcSubnet <ResourceGroup_Name>/<VirtualNetwork_Name>/<SubnetName> Subnet associated with the source IP in the flow.
DestSubnet <ResourceGroup_Name>/<VirtualNetwork_Name>/<SubnetName> Subnet associated with the destination IP in the flow.
SrcApplicationGateway <SubscriptionID>/<ResourceGroupName>/<ApplicationGatewayName> Application gateway associated with the source IP in the flow.
DestApplicationGateway <SubscriptionID>/<ResourceGroupName>/<ApplicationGatewayName> Application gateway associated with the destination IP in the flow.
SrcExpressRouteCircuit <SubscriptionID>/<ResourceGroupName>/<ExpressRouteCircuitName> ExpressRoute circuit ID - when flow is sent from site via ExpressRoute.
DestExpressRouteCircuit <SubscriptionID>/<ResourceGroupName>/<ExpressRouteCircuitName> ExpressRoute circuit ID - when flow is received from cloud by ExpressRoute.
ExpressRouteCircuitPeeringType - AzurePrivatePeering
- AzurePublicPeering
- MicrosoftPeering		 ExpressRoute peering type involved in the flow.
SrcLoadBalancer <SubscriptionID>/<ResourceGroupName>/<LoadBalancerName> Load balancer associated with the source IP in the flow.
DestLoadBalancer <SubscriptionID>/<ResourceGroupName>/<LoadBalancerName> Load balancer associated with the destination IP in the flow.
SrcLocalNetworkGateway <SubscriptionID>/<ResourceGroupName>/<LocalNetworkGatewayName> Local network gateway associated with the source IP in the flow.
DestLocalNetworkGateway <SubscriptionID>/<ResourceGroupName>/<LocalNetworkGatewayName> Local network gateway associated with the destination IP in the flow.
ConnectionType - VNetPeering
- VpnGateway
- ExpressRoute		 The connection type.
ConnectionName <SubscriptionID>/<ResourceGroupName>/<ConnectionName> The connection name. For flow type P2S, it's formatted as <GatewayName>_<VPNClientIP>
ConnectingVNets Space separated list of virtual network names. In hub and spoke topology, hub virtual networks are populated here.
Country Two-letter country code (ISO 3166-1 alpha-2) Populated for flow type ExternalPublic. All IP addresses in PublicIPs field share the same country code.
AzureRegion Azure region locations Populated for flow type AzurePublic. All IP addresses in PublicIPs field share the Azure region.
AllowedInFlows - Count of inbound flows that were allowed, which represents the number of flows that shared the same four-tuple inbound to the network interface at which the flow was captured.
DeniedInFlows - Count of inbound flows that were denied. (Inbound to the network interface at which the flow was captured).
AllowedOutFlows - Count of outbound flows that were allowed (Outbound to the network interface at which the flow was captured).
DeniedOutFlows - Count of outbound flows that were denied (Outbound to the network interface at which the flow was captured).
PacketsDestToSrc - Represents packets sent from the destination to the source of the flow.
PacketsSrcToDest - Represents packets sent from the source to the destination of the flow .
BytesDestToSrc - Represents bytes sent from the destination to the source of the flow.
BytesSrcToDest - Represents bytes sent from the source to the destination of the flow.
CompletedFlows - Total number of flows completed (populated with non-zero value when a flow gets a completed event).
SrcPublicIPs <SOURCE_PUBLIC_IP>|<FLOW_STARTED_COUNT>|<FLOW_ENDED_COUNT>|<OUTBOUND_PACKETS>|<INBOUND_PACKETS>|<OUTBOUND_BYTES>|<INBOUND_BYTES> Entries separated by bars.
DestPublicIPs <DESTINATION_PUBLIC_IP>|<FLOW_STARTED_COUNT>|<FLOW_ENDED_COUNT>|<OUTBOUND_PACKETS>|<INBOUND_PACKETS>|<OUTBOUND_BYTES>|<INBOUND_BYTES> Entries separated by bars.
FlowEncryption - Encrypted
- Unencrypted
- Unsupported hardware
- Software not ready
- Drop due to no encryption
- Discovery not supported
- Destination on same host
- Fall back to no encryption.		 Encryption level of flows.
PrivateEndpointResourceId <ResourceGroup/privateEndpointResource> Resource ID of the private endpoint resource. Populated when traffic is flowing to or from a private endpoint resource.
PrivateLinkResourceId <ResourceGroup/ResourceType/privateLinkResource> Resource ID of the private link service. Populated when traffic is flowing to or from a private endpoint resource.
PrivateLinkResourceName Plain text Resource name of the private link service. Populated when traffic is flowing to or from a private endpoint resource.
IsFlowCapturedAtUDRHop - True
- False		 If the flow was captured at a UDR hop, the value is True.

Note

NTANetAnalytics in virtual network flow logs replaces AzureNetworkAnalytics_CL used in network security group flow logs.

Public IP details schema

Traffic analytics provides WHOIS data and geographic location for all public IPs in your environment. For a malicious IP, traffic analytics provides DNS domain, threat type and thread descriptions as identified by Microsoft security intelligence solutions. IP Details are published to your Log Analytics workspace so you can create custom queries and put alerts on them. You can also access prepopulated queries from the traffic analytics dashboard.

The following table details public IP schema. For more information, see NTAIpDetails.

Field Format Comments
TableName NTAIpDetails Table that contains traffic analytics IP details data.
SubType FlowLog Subtype for the flow logs. Use only FlowLog. Other values of SubType are for internal workings of the product.
FASchemaVersion 2 Schema version. Doesn't reflect virtual network flow Log version.
FlowIntervalStartTime Date and time in UTC Start time of the flow log processing interval (the time from which flow interval is measured).
FlowIntervalEndTime Date and time in UTC End time of the flow log processing interval.
FlowType - AzurePublic
- ExternalPublic
- MaliciousFlow		 See Notes for definitions.
IP Public IP Public IP whose information is provided in the record.
PublicIPDetails Information about IP For AzurePublic IP: Azure Service owning the IP or Microsoft Virtual Public IP for the IP 168.63.129.16.
ExternalPublic/Malicious IP: WhoIS information of the IP.
ThreatType Threat posed by malicious IP For Malicious IPs only. One of the threats from the list of currently allowed values. For more information, see Notes.
DNSDomain DNS domain For Malicious IPs only. Domain name associated with this IP.
ThreatDescription Description of the threat For Malicious IPs only. Description of the threat posed by the malicious IP.
Location Location of the IP For Azure Public IP: Azure region of virtual network / network interface / virtual machine to which the IP belongs or Global for IP 168.63.129.16.
For External Public IP and Malicious IP: two-letter country code (ISO 3166-1 alpha-2) where IP is located.
Url URL corresponding to the malicious IP For Malicious IPs only.
Port Port corresponding to the malicious IP For Malicious IPs only.

Note

  • NTAIPDetails in virtual network flow logs replaces AzureNetworkAnalyticsIPDetails_CL used in network security group flow logs.

  • Traffic analytics can log any malicious FQDN associated to the IP for malicious flows. To filter out, use the port, URL and domain fields as needed.

Threat types

The following table lists the currently allowed values for the ThreatType field in the traffic analytics IP details schema.

Value Description
Botnet Indicator detailing a botnet node/member.
C2 Indicator detailing a Command & Control node of a botnet.
CryptoMining Traffic involving this network address / URL is an indication of CyrptoMining / Resource abuse.
DarkNet Indicator of a Darknet node/network.
DDoS Indicators relating to an active or upcoming DDoS campaign.
MaliciousUrl URL that is serving malware.
Malware Indicator describing a malicious file or files.
Phishing Indicators relating to a phishing campaign.
Proxy Indicator of a proxy service.
PUA Potentially Unwanted Application.
WatchList A generic bucket into which indicators are placed when it can't be determined exactly what the threat is or requires manual interpretation. WatchList should typically not be used by partners submitting data into the system.

Notes

  • In case of AzurePublic and ExternalPublic flows, customer owned Azure virtual machine IP is populated in VMIP_s field, while the Public IP addresses are populated in the PublicIPs_s field. For these two flow types, you should use VMIP_s and PublicIPs_s instead of SrcIP_s and DestIP_s fields. For AzurePublic and ExternalPublic IP addresses, we aggregate further, so that the number of records ingested to Log Analytics workspace is minimal. (This field will be deprecated. Use SrcIP_s and DestIP_s depending on whether the virtual machine was the source or the destination in the flow).
  • Some field names are appended with _s or _d, which don't signify source and destination but indicate the data types string and decimal respectively.
  • Based on the IP addresses involved in the flow, we categorize the flows into the following flow types:
    • IntraVNet: Both IP addresses in the flow reside in the same Azure virtual network.
    • InterVNet: IP addresses in the flow reside in two different Azure virtual networks.
    • S2S (Site-To-Site): One of the IP addresses belongs to an Azure virtual network, while the other IP address belongs to customer network (Site) connected to the virtual network through VPN gateway or ExpressRoute.
    • P2S (Point-To-Site): One of the IP addresses belongs to an Azure virtual network, while the other IP address belongs to customer network (Site) connected to the Azure Virtual Network through VPN gateway.
    • AzurePublic: One of the IP addresses belongs to an Azure virtual network, while the other IP address is an Azure Public IP address owned by Microsoft. Customer owned Public IP addresses aren't part of this flow type. For instance, any customer owned VM sending traffic to an Azure service (Storage endpoint) would be categorized under this flow type.
    • ExternalPublic: One of the IP addresses belongs to an Azure virtual network, while the other IP address is a public IP that is neither owned by Microsoft nor part of a customer-owned subscription visible to traffic analytics and isn't reported as malicious in the ASC feeds that traffic analytics consumes for the processing interval between FlowIntervalStartTime_t and FlowIntervalEndTime_t.
    • MaliciousFlow: One of the IP addresses belong to an Azure virtual network, while the other IP address is a public IP that is neither owned by Microsoft nor part of a customer-owned subscription visible to traffic analytics and is reported as malicious in the ASC feeds that traffic analytics consumes for the processing interval between FlowIntervalStartTime_t and FlowIntervalEndTime_t.
    • UnknownPrivate: One of the IP addresses belong to an Azure virtual network, while the other IP address belongs to the private IP range defined in RFC 1918 and couldn't be mapped by traffic analytics to a customer owned site or Azure virtual network.
    • Unknown: Unable to map either of the IP addresses in the flow with the customer topology in Azure and on-premises (site).

Note

A subscription is visible to traffic analytics in a Log Analytics workspace if it contains a flow log configured to that workspace.

Related content