Azure Arc
A Microsoft cloud service that enables deployment of Azure services across hybrid and multicloud environments.
525 questions
I wonder, do you do any packet inspection?
Azure Arc connected machine agent fails to connect with error: wsarecv: An existing connection was forcibly closed by the remote host.
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(121.6, 25%, 21%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMB%3C/text%3E%3C/svg%3E)
Maya Butt
45
Reputation points
When attempting to connect an onprem server to Azure ARC the process fails with an error:
(removed private info for security reasons)
ERROR Get "https://agentserviceapi.guestconfiguration.azure.com": read tcp <privateIP>:10091->13.66.149.68:443: wsarecv: An existing connection was forcibly closed by the remote host.
INFO Exit Code: AZCM0026: Network Error
INFO For troubleshooting, see https://aka.ms/arc/azcmerror
FATAL required endpoints unavailable: https://westus2-gas.guestconfiguration.azure.com,https://agentserviceapi.guestconfiguration.azure.com
Thinking this was a firewall issue, we had our network team investigate, however they are seeing the traffic going through to the internet from this machines IP, and being refused by the endpoint. There are multiple other servers on the same subnet that are able to connect with no issues. Network team assures me there is no special routing or rules for this particular client machine.
I ran the install again with the verbose logging and have the full output in a text file, but there isn't much more info that indicates what the problem might be. (attached in plaintext file)
I have also checked and made sure the service principal is correct and that the secret is not expired, and the other options such as resource group, tenant ID and subscription ID are all correct. Here is the command options I am using (again removed all the private information):
& "$env:ProgramFiles\AzureConnectedMachineAgent\azcmagent.exe" connect --resource-group "resource-group" --tenant-id "<tenant-id>" --location "westus2" --subscription-id "<subscription-id>" --cloud "AzureCloud" --tags "Datacenter=**" --correlation-id "correlation-id" --verbose
Called remotely via powershell, also tried running locally on the server, same result)
Additional Info:
Client is Windows Server 2019 VM running on VMware ESXi, 7.0.3
Azure connected machine agent is 1.25.02203.713
Any help or some clues where to look next would be greatly appreciated.
Azure Arc
{count} votes
-
Maya Butt 45 Reputation points
2023-02-24T15:10:13.7133333+00:00 Not yet, our network team thinks it may be related to TLS decryption and inspection happening at the firewall but I haven't been able to find any recommendations from MS that this traffic should not be inspected. We are also considering using a proxy or private endpoint instead of the public endpoint to get around this issue and others.
-
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(112.00000000000001, 68%, 20%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKS%3C/text%3E%3C/svg%3E)
Karuppiah Sathappan (DHL IT Services) 0 Reputation points2023-02-28T09:10:26.6+00:00 Have found any answer, I also got same issue.
-
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(275.2, 41%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAM%3C/text%3E%3C/svg%3E)
AnuragSingh-MSFT 21,551 Reputation points Moderator2023-02-28T09:15:06.5366667+00:00 @Maya Butt thank you for reaching out here. It seems that you are using the private link to connect On-Prem machine to Azure Arc. Please ensure that the steps mentioned in this link has been followed:
- Check your on-premises DNS server(s) to verify it is either forwarding to Azure DNS or is configured with appropriate A records in your private link zone. These lookup commands should return private IP addresses in your Azure virtual network. If they resolve public IP addresses, double check your machine or server and network's DNS configuration.
nslookup gbl.his.arc.azure.comnslookup agentserviceapi.guestconfiguration.azure.com - If you are having trouble onboarding a machine or server, confirm that you've added the Azure Active Directory and Azure Resource Manager service tags to your local network firewall. The agent needs to communicate with these services over the internet until private endpoints are available for these services. If you are still facing this issue, I would request reaching out to Azure Support using this link - https://azure.microsoft.com/en-us/support Based on the nature of the issue, it will require 1:1 investigation and trace analysis, which cannot be done through an online forum. If you are facing issues reaching out to Azure Support, please let us know.
- Check your on-premises DNS server(s) to verify it is either forwarding to Azure DNS or is configured with appropriate A records in your private link zone. These lookup commands should return private IP addresses in your Azure virtual network. If they resolve public IP addresses, double check your machine or server and network's DNS configuration.
-
AnuragSingh-MSFT 21,551 Reputation points Moderator
2023-03-02T08:52:33.19+00:00 @Maya Butt following up to check if you had a chance to review the reply above. Please let us know if you have any questions.
-
Maya Butt 45 Reputation points
2023-03-02T16:12:52.49+00:00 Sorry, I was waiting for something our network team was working on to see if that helped before replying or opening a support case. Seems that there is some https interception\decryption going on at the firewall for this server that I suspect is causing issues. Once I have more information and a chance to test after they make some changes I will provide an update.
-
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(268.8, 99%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EBM%3C/text%3E%3C/svg%3E)
Burgess, Martin 5 Reputation points2023-03-02T17:13:46.2333333+00:00 I have the same issue, however, I was able to install it on a couple other servers first. Then I tried installing it on a third server and I got this error. Same firewall in front of all servers so I'm not sure that is the issue for me.
Sign in to comment
Accepted answer
-
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(192, 2%, 28%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EPJ%3C/text%3E%3C/svg%3E)
Peter Jensen (Insight Global, Inc) 80 Reputation points2023-06-08T14:14:10.23+00:00 -
Maya Butt 45 Reputation points
2023-06-08T14:17:01.73+00:00 Turns out we do, and this was exactly the problem.
-
Peter Jensen (Insight Global, Inc) 80 Reputation points
2023-06-08T14:19:41.19+00:00 Glad it helped!
-
Peter Jensen (Insight Global, Inc) 80 Reputation points
2023-06-08T14:20:38.72+00:00 Did your team simply not continue packet inspection, or did you do anything else along the way?
-
Maya Butt 45 Reputation points
2023-06-08T14:25:35.0566667+00:00 We ended up excluding the server that was having issues from the packet inspection policy. We are investigating using a private endpoint for ARC instead to resolve this on the remaining servers that are having issues connecting.
-
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(25.6, 22%, 12%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EGS%3C/text%3E%3C/svg%3E)
Geevee Sajan T 0 Reputation points2023-06-26T12:34:35.1633333+00:00 Hey Jaime,
i am facing the same issue, any good news from your side?
Sign in to comment -