Azure Monitor Agent unable to communicate with Log Analytics Workspace; During Install Unable to find the arc token directory; errorcode 2148795139

Peter Smit 1

Clean installation of the Arc Agent, afterwards via DCR a Azure Monitor Agent was deployed to the ubuntu box

LinuxAMA does not communicate since Refreshtoken cannot be found:

/var/opt/microsoft/azuremonitoragent/log/mdsd.err >> GetMSI token from Arc IMDS: Retryable Exeption - Unable to find the arc token directory. Errorcode:2148795139

any thoughts?

2 answers

AnuragSingh-MSFT 19,686 Reputation points

2021-10-21T06:35:42.433+00:00
Hi @Peter Smit , apologies for the delayed response. Can you please check the following and ensure that these are in order?

Please connect to this Ubuntu machine and run the command below:
sudo azcmagent show

Ensure that the "Agent Status" field is "Connected" in the result from previous step. If this is not connected, you can run the azcmagent connect command to ensure that the arc agent is connected. (This might require to be run as sudo). The parameters for this command can be obtained from the onboarding script from Azure portal from "Azure Arc --> Servers\Add.." option.

For the VM resource created in portal for this machine, please check the status of "AzureMonitorLinuxAgent" extension. For successful install, it should be "Succeeded".

Ensure that the service "azuremonitoragent" on the machine is in running status (service azuremonitoragent status)

In case you are still facing this issue after ensuring that the above points are all OK, please share details about this machine:

OS version (please refer here for supported OS for the agent)

version of Azure Monitoring agent installed ( apt list --installed | grep azuremonitoragent )

Are there other Linux/Ubuntu machines where this works?
Please sign in to rate this answer.
AnuragSingh-MSFT 19,686 Reputation points

2021-10-26T06:22:48.04+00:00

@Peter Smit , I wanted to check if you had a chance to review my answer above. Please let me know if you have any queries or concerns.
Please 'Accept as answer' if it helped so that it can help others in the community looking for help on similar topics.

Peter Smit 1 Reputation point

2021-10-26T06:55:01.657+00:00

sudo azcmagent show gives me an overview of an connected ARC agent with an agent status "Connected"
<redacted/>

details machine:
<redacted/>

Extension Details:
<redacted/>

I have another test linux machine which is connected with the Log Analytics Monitor Agent which communincates fine with Log Analytics.

Peter Smit 1 Reputation point

2021-10-26T06:55:57.733+00:00

apologies for my late response, I was on leave..

Peter Smit 1 Reputation point

2021-10-26T08:31:09.147+00:00

After removing and re-assigning the extension via DCR, the extension at first gives a status Enable succeeded.. after a while it will give the error as displayed in previous images.

AnuragSingh-MSFT 19,686 Reputation points

2021-10-26T10:04:53.303+00:00

Thank you @Peter Smit . To be sure, can you please confirm if the following steps were followed (if not, can you please perform them and share the result)

Remove this source from the DCR (Please go to the DCR associated with this VM in Azure Monitor --> Resources --> "delete" this VM from here*. This step will not remove the agent or extension from the VM).

Uninstall the AzureMonitorLinuxAgent extension from this VM through Azure portal (under "Extension" blade of the VM)

Ensure that azuremonitoragent package is not available on the machine, after step 2 above completes.

Retry agent installation by adding this VM to DCR as a source.

In case the steps above do not work, I would suggest opening a Support Ticket with Microsoft as this may require additional troubleshooting. Please let me know if you have any questions.

Peter Smit 1 Reputation point

2021-10-26T10:50:39.597+00:00

i found some logging in /var/opt/microsoft/azuremonitoragent/log/mdsd.err and that log is full with the message

Failed to get MSI token from Arc IMDS: Retryable Exception - Unable to find the arc token directory. Errorcode:2148795139

AnuragSingh-MSFT 19,686 Reputation points

2021-10-26T13:40:31.877+00:00

Thank you @Peter Smit . Please give me some time to look in this error and I will get back to you.

Also, I am sorry if I missed it, but can you please confirm if you had followed the steps shared above to see if it resolves the issue. (I am trying to get this done because it looks like the agent was not installed properly because of the status message on portal, which might be causing the IMDS error. The error mentioned should be triggered after the agent has been installed when it tries to authenticate)

Please let me know if you have any questions.

AnuragSingh-MSFT 19,686 Reputation points

2021-11-03T05:07:09.5+00:00

@Peter Smit , I wanted to update you that we are investigating the issue internally and will get back on this thread with further update.
Sign in to comment
Shayoni Seth 1 Reputation point Microsoft Employee

2021-11-05T18:18:49.4+00:00

@Peter Smit , As now documented, AMA Linux 1.12.2.0 has a regression on Arc machines. Rollout of next version 1.14 is in progress, updating to this new or previous version should work on Arc.
https://learn.microsoft.com/en-us/azure/azure-monitor/agents/azure-monitor-agent-install?tabs=ARMAgentPowerShell%2CPowerShellWindows%2CPowerShellWindowsArc%2CCLIWindows%2CCLIWindowsArc#extension-versions
Please sign in to rate this answer.
AnuragSingh-MSFT 19,686 Reputation points

2021-11-10T03:19:00.927+00:00

@Peter Smit , I wanted to check if you had a chance to review the answers above. Please let us know if you have any queries or concerns.
Please 'Accept as answer' if it helped so that it can help others in the community looking for help on similar topics.
Sign in to comment