Monitoring On-premises servers using Azure monitor

Question

Monitoring On-premises servers using Azure monitor

krupa gundraju 45

Hi Team,

We are trying to monitor on-prem servers using Azure, For that I installed azure connected machine agent, and I could see the machines are register with Azure arc too but when we see the performance information there "error retrieving data".

User's image

I verified from our side the traffic is flowing from on-prem servers IP to below destinations at port 443

Dest:

management.azure.com

*.his.arc.azure.com

*.guestconfiguration.azure.com

And we created the script by selecting private endpoint while generating script from Azure arc. Are we missing anything?

Thanks

Krupa G

krupa gundraju 45 Reputation points

2025-04-04T10:25:10.6633333+00:00

Hi Stanislav,

We followed all these steps, but it is not able to retrieve the data.

While creating the script for azure arc registration we selected to use private endpoint.

As you can see in my first screenshot the server itself is not connected to azure post installing the 'azure connected machine agent'.

and This is the first we are using azure arc and see no way to find logs too.

where can we get troubleshooting logs for this connectivity issue.?

Thanks

Krupa G
Stanislav Zhelyazkov 28,186 Reputation points MVP Volunteer Moderator

2025-04-04T10:35:15.66+00:00

Ok. I have missed the private endpoint stuff. You should check if you have network connectivity from your server to Azure Monitor.
Stanislav Zhelyazkov 28,186 Reputation points MVP Volunteer Moderator

2025-04-04T10:44:05.5133333+00:00

Additionally check this troubleshooting steps. It sounds more like you have some general networking issue and this should be tagged to either Azure Arc or Azure networking rather Azure Monitor.
krupa gundraju 45 Reputation points

2025-04-07T16:41:41.7866667+00:00

Hi @Stanislav Zhelyazkov

We have below error when we try to install connected machine agent, we are trying to connect via PE but seems it is picking up public endpoint

Error

Installation of azcmagent completed successfully

time="2025-04-07T09:49:05-04:00" level=info msg="Connecting machine to Azure... This might take a few minutes."

time="2025-04-07T09:49:06-04:00" level=info msg="Testing connectivity to endpoints that are needed to connect to Azure... This might t

ake a few minutes."

time="2025-04-07T09:49:08-04:00" level=warning msg="Expect agentserviceapi.guestconfiguration.azure.com to resolve to private IP addre

sses but got one or more public IP addresses: xxx, xxx"

time="2025-04-07T09:49:08-04:00" level=warning msg="Expect gbl.his.arc.azure.com to resolve to private IP addresses but got one or mor

e public IP addresses: xxx, xxx"

time="2025-04-07T09:49:08-04:00" level=warning msg="Expect eastus-gas.guestconfiguration.azure.com to resolve to private IP addresses

but got one or more public IP addresses: xxx, xxx"

time="2025-04-07T09:49:08-04:00" level=warning msg="Expect gbl.his.arc.azure.com to resolve to private IP addresses but got one or mor

e public IP addresses: xxxx, xxx"

20% [[32m=[0m[32m=[0m[32m>[0m ]

[0m

30% [[32m=[0m[32m=[0m[32m=[0m[32m>[0m ]

[0mtime="2025-04-07T09:49:14-04:00" level=info msg="Creating resource in Azure..." Correlation ID=xxxx Resource ID=xxxx

60% [[32m=[0m[32m=[0m[32m=[0m[32m=[0m[32m=[0m[32m=[0m[32m=[0m[32m=[0m[32m>[0m ]

[0mtime="2025-04-07T09:50:18-04:00" level=error msg="Failed to get MSI certificate. You may delete the ARM resource and run azcmage

nt connect to try again" Attempts=10 type=Security

time="2025-04-07T09:50:18-04:00" level=error msg="Created Arc resource xxx but couldn't connect it. You may delete the ARM resou

rce and run azcmagent connect again to onboard."

time="2025-04-07T09:50:19-04:00" level=info msg="AZCM0081: Failed to Get MSI Certificate from HIS"

time="2025-04-07T09:50:19-04:00" level=info msg="Please contact Microsoft Support for assistance."

time="2025-04-07T09:50:19-04:00" level=info msg="For more troubleshooting tips, please refer to https://aka.ms/arc/azcmerror"

time="2025-04-07T09:50:19-04:00" level=fatal msg="failed to retrieve MSI cert: 403 Forbidden{"error":{"code":"HCRP403","message

":"Access to the specified resource from Public Internet is disallowed. [VmId=\xxx\u0022]",

"target":"/machine/xxx/identity"}}"
Stanislav Zhelyazkov 28,186 Reputation points MVP Volunteer Moderator

2025-04-08T05:37:12.9366667+00:00

I will re-tag your issue to Azure Arc and convert the existing answer to replies so someone else can pick the issue due to the connectivity of Azure Arc agent is out of my scope.

May be your traffic to public internet is not allowed and considering this note from the official documentation:

"Network traffic from the Azure Connected Machine agent to Microsoft Entra ID (login.windows.net, login.microsoftonline.com, pas.windows.net) and Azure Resource Manager (management.azure.com) will continue to use public endpoints. If your server needs to communicate through a proxy server to reach these endpoints, configure the agent with the proxy server URL before connecting it to Azure. You might also need to configure a proxy bypass for the Azure Arc services if your private endpoint is not accessible from your proxy server."

is not taken into account.
krupa gundraju 45 Reputation points

2025-04-08T09:50:09.1533333+00:00

Hi,

Issue: 1

This issue was because Azure arc private endpoints got private dns zone created and record sets too but our internal VNET dns server is custom server, post deleting private dns zones and recreating dns in custom dns servers it started working.

Issue 2:

The traffic was blocking in the internal SDWAN device.

Thanks

Krupa G

Accepted answer

1 additional answer

Your answer

krupa gundraju 45 Reputation points

2025-04-04T10:25:10.6633333+00:00

Hi Stanislav,

We followed all these steps, but it is not able to retrieve the data.

While creating the script for azure arc registration we selected to use private endpoint.

As you can see in my first screenshot the server itself is not connected to azure post installing the 'azure connected machine agent'.

and This is the first we are using azure arc and see no way to find logs too.

where can we get troubleshooting logs for this connectivity issue.?

Thanks

Krupa G
Stanislav Zhelyazkov 28,186 Reputation points MVP Volunteer Moderator

2025-04-04T10:35:15.66+00:00

Ok. I have missed the private endpoint stuff. You should check if you have network connectivity from your server to Azure Monitor.
Stanislav Zhelyazkov 28,186 Reputation points MVP Volunteer Moderator

2025-04-04T10:44:05.5133333+00:00

Additionally check this troubleshooting steps. It sounds more like you have some general networking issue and this should be tagged to either Azure Arc or Azure networking rather Azure Monitor.
Stanislav Zhelyazkov 28,186 Reputation points MVP Volunteer Moderator

2025-04-08T05:37:12.9366667+00:00

I will re-tag your issue to Azure Arc and convert the existing answer to replies so someone else can pick the issue due to the connectivity of Azure Arc agent is out of my scope.

May be your traffic to public internet is not allowed and considering this note from the official documentation:

"Network traffic from the Azure Connected Machine agent to Microsoft Entra ID (login.windows.net, login.microsoftonline.com, pas.windows.net) and Azure Resource Manager (management.azure.com) will continue to use public endpoints. If your server needs to communicate through a proxy server to reach these endpoints, configure the agent with the proxy server URL before connecting it to Azure. You might also need to configure a proxy bypass for the Azure Arc services if your private endpoint is not accessible from your proxy server."

is not taken into account.
krupa gundraju 45 Reputation points

2025-04-08T09:50:09.1533333+00:00

Hi,

Issue: 1

This issue was because Azure arc private endpoints got private dns zone created and record sets too but our internal VNET dns server is custom server, post deleting private dns zones and recreating dns in custom dns servers it started working.

Issue 2:

The traffic was blocking in the internal SDWAN device.

Thanks

Krupa G

Answer 1

Hi @krupa gundraju

I'm glad that you were able to resolve your issue and thank you for posting your solution so that others experiencing the same thing can easily reference this! Since the Microsoft Q&A community has a policy that "The question author cannot accept their own answer. They can only accept answers by others ", I'll repost your solution in case you'd like to "Accept " the answer.

Issue: Monitoring On-premises servers using Azure monitor

Solution:

The issue occurred because Azure Arc private endpoints created private DNS zones incompatible with our custom DNS server, and traffic was also blocked by an internal SD-WAN device; it was resolved by updating DNS records on the custom server and unblocking the traffic

Please remember to "Accept Answer" if any answer/reply helped, so that others in the community facing similar issues can easily find the solution.

Accept answers on Microsoft Q&A | Microsoft Learn

An accepted answer is the answer that the person who asked the question chooses as the one they think best solves their problem.

Answer 2

Stanislav Zhelyazkov 28,186 MVP Volunteer Moderator

Hi,

Onboarding to Azure Arc for servers does not automatically enables monitoring. You can see full perquisites of enabling VM insights at: Enable VM Insights. At the end your ARC servers need Azure Monitor agent and dependency agent extensions installed and the data collection rule to be associated with your Arc servers.

Please "Accept the answer" if the information helped you. This will help us and others in the community as well.

Share via

Monitoring On-premises servers using Azure monitor

1 additional answer

Your answer