Configure and troubleshoot non-Azure monitoring sources in Network Watcher Connection Monitors

Question

I am trying to use non-Azure endpoints (specifically Azure Arc servers) as Connection Monitor monitoring sources in Network Watcher.

Connection Monitors with Azure VM monitoring sources are working as expected, but Connection Monitors configured with non-Azure monitoring sources fail to return a Pass status, even after waiting for 3 days they still display Indeterminate status.

I'd like guidance on the requirements for using non-Azure endpoints as monitoring sources in Network Watcher Connection Monitors and on how data flows from those agents into the Log Analytics workspace.

Regarding requirements for using non-Azure endpoints:

This document guides that the Log Analytics agent and Network Performance Monitor solution are required for Connection Monitors with on-premises machines. I followed this guidance, inlcuding running the EnableRules.ps1 script. It seems that this documentation is outdated, but also if the Network Performance Monitor solution is not enabled on this Log Analytics workspace, then an error is presented when creating Connection Monitors using non-Azure endpoint sources.

This document guides that you can use the Azure Monitor agent and do not need the Log Analytics agent and then has the following related document providing install guidance for Azure Connected Machine agent, Azure Monitor Agent, and Network Watcher agent, and guiding the user to Enable the Network Watcher agent using the New-AzConnectedMachineExtension cmdlet.

Currently we have a mix of configurations on the non-Azure test systems (some with Log Analytics agent/Azure Monitor Agent/Network Watcher and some with only Azure Monitor Agent/Network Watcher Agent combo), and we have not been able to get any of the connection monitors configured using non-Azure endpoint sources to show any status other than Indeterminate.

What components are needed for using non-Azure endpoints as monitoring sources in Network Watcher Connection Monitors?

Additionally, how do Test results flow from an agent and into the Log Analytics Workspace Logs?

In the agent logs of non-Azure test systems (C:\ProgramData\GuestConfig\extension_logs\Microsoft.Azure.NetworkWatcher.NetworkWatcherAgentWindows) we see that the systems are receiving the Test configurations/parameters from Network Watcher and executing them. But also there are the following Error messages, which don't appear in the logs of the properly-functioning Azure VMs. Do these errors have significance?

	Line 35974: U07/17/23 14:22:38:294 T18e0     E NetAgent::CommsTransportAMA::CreateFinalResponsesAndUploadDataToAMA: Unable to fetch AMARespones from received NMMessage.
	Line 36170: U07/17/23 14:25:43:887 T18e0     E NetAgent::CommsTransportAMA::HandleCleanupConnectivityCheckResponse: Unable to obtain StartConnectivityCheck messages from AMAConfigManager for CMResourceId: /subscriptions/<my-subscription-id>/resourceGroups/networkwatcherrg/providers/Microsoft.Network/networkWatchers/NetworkWatcher_eastus/connectionMonitors/<connection-monitor-name>.
	Line 36289: U07/17/23 14:30:43:277 T18e0     E NetAgent::CommsTransportAMA::HandleCleanupConnectivityCheckResponse: Unable to obtain StartConnectivityCheck messages from AMAConfigManager for CMResourceId: /subscriptions/<my-subscription-id>/resourceGroups/networkwatcherrg/providers/Microsoft.Network/networkWatchers/NetworkWatcher_eastus/connectionMonitors/<connection-monitor-name>.

I also see there are Data Collection Rules created in Azure Monitor that correspond to Network Watcher Connection Monitors that use non-Azure endpoint monitoring sources (named Microsoft-NetworkWatcher-<test-name>). Is there any verification we can do on them?

What Log table will the test results appear in for non-Azure endpoints? The same NWConnectionMonitorTestResult table as the Azure endpoint tests or the NetworkMonitoring table or some other table?

Additionally, could this specific Network Watcher instance be broken? Is removing and re-adding the Network Watcher a valid troubleshooting step? This is not preferable because we have functional Connection Monitors in place, should we consider that while troubleshooting this problem.

Please let me know what additional information I can provide.

Answer 1

For us, the issue was related to using version 1.16 of the Azure Monitor Agent extension. The issue was resolved when we updated to version 1.18. While waiting for that to be released, we were able to downgrade to version 1.14 to work around the issue.

It's worth noting that the Network Watcher Agent service, running inside the operating system of an affected Arc-enabled server, needed to be restarted anytime the Azure Monitor Agent on that system was upgraded or downgraded.