Agent offline: Failed to establish IPC: open \\\\.\\PIPE\\himds: The system cannot find the file specified.

Question

Agent offline: Failed to establish IPC: open \\\\.\\PIPE\\himds: The system cannot find the file specified.

luuk jansen 5

Getting the [Title ] error in the azcmagent.log

The server status shows offline.

Restarted the HIMDS service.

Commands like azcmagent show, check don't work. Same error messages as in the title.

The agentconfig.json and localconfig.json files are there in the config folder in programdata > azureconnectedmachineagent.

AnuragSingh-MSFT 21,551 Reputation points Moderator

2023-03-10T08:23:59.7766667+00:00

luuk jansen, following up to check if you had a chance to review my answer below.

Please let me know if you have any questions. If the answer helped, please click Accept answer so that it can help others in the community looking for help on similar topics.

2 answers

Your answer

AnuragSingh-MSFT 21,551 Reputation points Moderator

2023-03-10T08:23:59.7766667+00:00

luuk jansen, following up to check if you had a chance to review my answer below.

Please let me know if you have any questions. If the answer helped, please click Accept answer so that it can help others in the community looking for help on similar topics.

Answer 1

AnuragSingh-MSFT 21,551 Moderator

luuk jansen, thank you for posting this question.

I understand that you have installed Arc agent on a Windows VM and it is working as expected. Here are some of the troubleshooting steps that should help isolate/troubleshoot the issue further

As the error states, the HIMDS pipe has not being found by the agent. This could be if the HIMDS agent is not running, it was unable to register for pipe OR if something is blocking the communication (like a local antivirus application). Check if the HIMDS service "Azure Hybrid Instance Metadata Service" is running. I see that you have attempted restarting this service, but does it stay running? You may also check the Application and System logs to understand if there are errors after starting/restarting the HIMDS service. Group polies may also impact the run of service
The following troubleshooting steps are applicable for all types of extensions, including Arc - general troubleshooting. Check to see if there are errors in these logs related to incomplete or interrupted onboarding of the agent.
If this issue is happening only on a single machine, it could mean that the machine is impacted. However, if there are multiple machines with similar issues, most probably it could be related to the domain policy or the security softwares used in the environment.
Finally, uninstalling the arc agent and reinstalling can also be attempted - uninstall the arc agent.

Please let us know if you have any questions.

If the answer helped, please click Accept answer so that it can help others in the community looking for help on similar topics.

luuk jansen 5 Reputation points

2023-03-14T10:53:14.4033333+00:00

Hello AnuragSingh-MSFT,

To answer your questions

The service is indeed running. When I reinstall the agent. The agent works for a couple of days and then they go offline. As of now I have 3 endpoints with this problem. 2/3 having this "agent offline" situation more recently. I followed the troubleshooting steps and checked the logs.

I'm finding the following logs:

himds.log

time="2023-03-06T16:46:04+01:00" level=warning msg="Service not available. Check Agent log for details." reason=service_unavailable

Azcmagent.log

time="2023-03-14T10:55:58+01:00" level=debug msg="Starting client connection on: \\.\PIPE\himds"

time="2023-03-14T10:55:58+01:00" level=debug msg="Failed to establish IPC: open \\.\PIPE\himds: The system cannot find the file specified....Retrying #1"

extensionHealth.log

[QueryIMDSEndpoint] ErrorCode:1601 ERROR: IMDS appears to not be present on this machine. Error: 12029; Message: A connection with the server could not be established
Ruben van Osch 106 Reputation points

2023-03-15T10:47:32.18+00:00

Hi AnuragSingh-MSFT,

I am a colleague of Luuk and found something interesting. I believe the HIMDS service shows it is running because it is running on a different server. I’m not too familiar with Domain servers, but if I look at the properties of the service I can see the following title: “Azure Hybrid Instance Metadata Service Properties (DomainServer02)”. The issue is currently happening on DomainServer01. Furthermore, we also had this problem on the DomainServer02. We solved that issue by reinstalling the agent. Now I assume it has to do with some sort of connection between the two Domain Servers.

(EDIT)
Never mind, I was looking at the wrong services view. The actual problem is that the "NT SERVICE\himds" account fails to logon with: "The user has not been granted the requested logon type at this machine."

Answer 2

Ruben van Osch 106

Due to a few oversights we eventually narrowed the problem down to the same issue described in the following question

The sollution then is to modify the Log on as a Service GPO by adding the NT SERVICE\himds account.

Share via

Agent offline: Failed to establish IPC: open \\\\.\\PIPE\\himds: The system cannot find the file specified.

2 answers

Your answer