Connection troubleshoot overview

With the increase of sophisticated and high-performance workloads in Azure, there's a critical need for increased visibility and control over the operational state of complex networks running these workloads. Such complex networks are implemented using network security groups, firewalls, user-defined routes, and resources provided by Azure. Complex configurations make troubleshooting connectivity issues challenging.

The connection troubleshoot feature of Azure Network Watcher helps reduce the amount of time to diagnose and troubleshoot network connectivity issues. The results returned can provide insights about the root cause of the connectivity problem and whether it's due to a platform or user configuration issue.

Connection troubleshoot reduces the Mean Time To Resolution (MTTR) by providing a comprehensive method of performing all connection major checks to detect issues pertaining to network security groups, user-defined routes, and blocked ports. It provides the following results with actionable insights where a step-by-step guide or corresponding documentation is provided for faster resolution:

  • Connectivity test with different destination types (VM, URI, FQDN, or IP Address)
  • Configuration issues that impact reachability
  • All possible hop by hop paths from the source to destination
  • Hop by hop latency
  • Latency (minimum, maximum, and average between source and destination)
  • Graphical topology view from source to destination
  • Number of probes failed during the connection troubleshoot check

Supported source and destination types

Connection troubleshoot provides the capability to check TCP or ICMP connections from any of these Azure resources:

  • Virtual machines
  • Virtual machine scale sets
  • Azure Bastion instances
  • Application gateways (except v1)

Important

Connection troubleshoot requires that the virtual machine you troubleshoot from has the Network Watcher agent VM extension installed. The extension is not required on the destination virtual machine.

Connection troubleshoot can test connections to any of these destinations:

  • Virtual machines
  • Fully qualified domain names (FQDNs)
  • Uniform resource identifiers (URIs)
  • IP addresses

Issues detected by connection troubleshoot

Connection troubleshoot can detect the following types of issues that can impact connectivity:

  • High VM CPU utilization
  • High VM memory utilization
  • Virtual machine (guest) firewall rules blocking traffic
  • DNS resolution failures
  • Misconfigured or missing routes
  • Network security group (NSG) rules that are blocking traffic
  • Inability to open a socket at the specified source port
  • Missing address resolution protocol entries for Azure ExpressRoute circuits
  • Servers not listening on designated destination ports

Response

The following table shows the properties returned after running connection troubleshoot.

Property Description
ConnectionStatus The status of the connectivity check. Possible results are Reachable and Unreachable.
AvgLatencyInMs Average latency during the connectivity check, in milliseconds. (Only shown if check status is reachable).
MinLatencyInMs Minimum latency during the connectivity check, in milliseconds. (Only shown if check status is reachable).
MaxLatencyInMs Maximum latency during the connectivity check, in milliseconds. (Only shown if check status is reachable).
ProbesSent Number of probes sent during the check. Maximum value is 100.
ProbesFailed Number of probes that failed during the check. Maximum value is 100.
Hops Hop by hop path from source to destination.
Hops[].Type Type of resource. Possible values are: Source, VirtualAppliance, VnetLocal, and Internet.
Hops[].Id Unique identifier of the hop.
Hops[].Address IP address of the hop.
Hops[].ResourceId Resource ID of the hop if the hop is an Azure resource. If it's an internet resource, ResourceID is Internet.
Hops[].NextHopIds The unique identifier of the next hop taken.
Hops[].Issues A collection of issues that were encountered during the check of the hop. If there were no issues, the value is blank.
Hops[].Issues[].Origin At the current hop, where issue occurred. Possible values are:
Inbound - Issue is on the link from the previous hop to the current hop.
Outbound - Issue is on the link from the current hop to the next hop.
Local - Issue is on the current hop.
Hops[].Issues[].Severity The severity of the detected issue. Possible values are: Error and Warning.
Hops[].Issues[].Type The type of the detected issue. Possible values are:
CPU
Memory
GuestFirewall
DnsResolution
NetworkSecurityRule
UserDefinedRoute
Hops[].Issues[].Context Details regarding the detected issue.
Hops[].Issues[].Context[].key Key of the key value pair returned.
Hops[].Issues[].Context[].value Value of the key value pair returned.
NextHopAnalysis.NextHopType The type of next hop. Possible values are:
HyperNetGateway
Internet
None
VirtualAppliance
VirtualNetworkGateway
VnetLocal
NextHopAnalysis.NextHopIpAddress IP address of next hop.
The resource identifier of the route table associated with the route being returned. If the returned route doesn't correspond to any user created routes, then this field will be the string System Route.
SourceSecurityRuleAnalysis.Results[].Profile Network configuration diagnostic profile.
SourceSecurityRuleAnalysis.Results[].Profile.Source Traffic source. Possible values are: *, IP Address/CIDR, and Service Tag.
SourceSecurityRuleAnalysis.Results[].Profile.Destination Traffic destination. Possible values are: *, IP Address/CIDR, and Service Tag.
SourceSecurityRuleAnalysis.Results[].Profile.DestinationPort Traffic destination port. Possible values are: * and a single port in the (0 - 65535) range.
SourceSecurityRuleAnalysis.Results[].Profile.Protocol Protocol to be verified. Possible values are: *, TCP and UDP.
SourceSecurityRuleAnalysis.Results[].Profile.Direction The direction of the traffic. Possible values are: Outbound and Inbound.
SourceSecurityRuleAnalysis.Results[].NetworkSecurityGroupResult Network security group result.
SourceSecurityRuleAnalysis.Results[].NetworkSecurityGroupResult.EvaluatedSecurityGroups[] List of results network security groups diagnostic.
SourceSecurityRuleAnalysis.Results[].NetworkSecurityGroupResult.SecurityRuleAccessResult The network traffic is allowed or denied. Possible values are: Allow and Deny.
SourceSecurityRuleAnalysis.Results[].NetworkSecurityGroupResult.EvaluatedSecurityGroups[].AppliedTo Resource ID of the NIC or subnet to which network security group is applied.
SourceSecurityRuleAnalysis.Results[].NetworkSecurityGroupResult.EvaluatedSecurityGroups[].MatchedRule Matched network security rule.
SourceSecurityRuleAnalysis.Results[].NetworkSecurityGroupResult.EvaluatedSecurityGroups[].MatchedRule.Action The network traffic is allowed or denied. Possible values are: Allow and Deny.
SourceSecurityRuleAnalysis.Results[].NetworkSecurityGroupResult.EvaluatedSecurityGroups[].MatchedRule.RuleName Name of the matched network security rule.
SourceSecurityRuleAnalysis.Results[].NetworkSecurityGroupResult.EvaluatedSecurityGroups[].NetworkSecurityGroupId Network security group ID.
SourceSecurityRuleAnalysis.Results[].NetworkSecurityGroupResult.RulesEvaluationResult[] List of network security rules evaluation results.
SourceSecurityRuleAnalysis.Results[].NetworkSecurityGroupResult.RulesEvaluationResult[].DestinationMatched Value indicates if destination is matched. Boolean values.
SourceSecurityRuleAnalysis.Results[].NetworkSecurityGroupResult.RulesEvaluationResult[].DestinationPortMatched Value indicates if destination port is matched. Boolean values.
SourceSecurityRuleAnalysis.Results[].NetworkSecurityGroupResult.RulesEvaluationResult[].Name Name of the network security rule.
SourceSecurityRuleAnalysis.Results[].NetworkSecurityGroupResult.RulesEvaluationResult[].ProtocolMatched Value indicates if protocol is matched. Boolean values.
SourceSecurityRuleAnalysis.Results[].NetworkSecurityGroupResult.RulesEvaluationResult[].SourceMatched Value indicates if source is matched. Boolean values.
SourceSecurityRuleAnalysis.Results[].NetworkSecurityGroupResult.RulesEvaluationResult[].SourcePortMatched Value indicates if source port is matched. Boolean values.
DestinationSecurityRuleAnalysis Same as SourceSecurityRuleAnalysis format.
SourcePortStatus Determines whether the port at source is reachable or not. Possible Values are:
Unknown
Reachable
Unstable
NoConnection
Timeout
DestinationPortStatus Determines whether the port at destination is reachable or not. Possible Values are:
Unknown
Reachable
Unstable
NoConnection
Timeout

The following example shows an issue found on a hop.

"Issues": [
    {
        "Origin": "Outbound",
        "Severity": "Error",
        "Type": "NetworkSecurityRule",
        "Context": [
            {
                "key": "RuleName",
                "value": "UserRule_Port80"
            }
        ]
    }
]

Fault types

Connection troubleshoot returns fault types about the connection. The following table provides a list of the possible returned fault types.

Type Description
CPU High CPU utilization.
Memory High Memory utilization.
GuestFirewall Traffic is blocked due to a virtual machine firewall configuration.

A TCP ping is a unique use case in which, if there's no allowed rule, the firewall itself responds to the client's TCP ping request even though the TCP ping doesn't reach the target IP address/FQDN. This event isn't logged. If there's a network rule that allows access to the target IP address/FQDN, the ping request reaches the target server and its response is relayed back to the client. This event is logged in the network rules log.
DNSResolution DNS resolution failed for the destination address.
NetworkSecurityRule Traffic is blocked by a network security group rule (security rule is returned).
UserDefinedRoute Traffic is dropped due to a user defined or system route.

Next step

To learn how to use connection troubleshoot to test and troubleshoot connections, continue to: