Azure Monitor
An Azure service that is used to collect, analyze, and act on telemetry data from Azure and on-premises environments.
Heartbeat data missing some data .error -Microsoft.EnterpriseManagement.HealthService.HealthAssessmentData.HealthAssessmentDataSource" reported an exception System.Net.WebException: The remote server returned an error: (400) Bad Request
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(92.8, 75%, 18%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EC%3C/text%3E%3C/svg%3E)
coolqwerty
61
Reputation points
Some of the heartbeat data is missing such as subscription ID . In the operation manager logs i am seeing below error which is the real cause of the problem - In below error is not able to connect with IMDS which has IP 169.254.169.254.
The establish the connectivity we have added the route so that VM IP can directly connect with 169.254.169.254 by bypassing the proxy but still getting 400 bad request.
Does anyone know solution of this ?
===============================================
Microsoft.EnterpriseManagement.HealthService.HealthAssessmentData.HealthAssessmentDataSource" reported an exception System.Net.WebException: The remote server returned an error: (400) Bad Request.
at System.Net.HttpWebRequest.GetResponse()
at Microsoft.EnterpriseManagement.HealthService.HealthAssessmentData.HealthAssessmentDataSource.TryGetIMDSData(Object state)
Azure Monitor
Azure Virtual Machines
Azure Virtual Machines
An Azure service that is used to provision Windows and Linux virtual machines.
{count} votes
-
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(275.2, 41%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAM%3C/text%3E%3C/svg%3E)
AnuragSingh-MSFT 21,566 Reputation points Moderator2024-03-19T06:24:58.4266667+00:00 coolqwerty, thank you for posting this question on Microsoft Q&A. Could you please follow the steps below/answer the queries to help us narrow down the issue further:
- You mentioned that you have seen this error in the "Operations Manager" logs. Are you using LA agent on this VM and not the AMA agent?
- How frequently do you see this error? The heartbeats are triggered every 1 minute, if the frequency of this error varies, it could mean intermittent issue on the machine itself.
- IMDS is a REST API that's available at a well-known, non-routable IP address (
169.254.169.254). You can only access it from within the VM. Communication between the VM and IMDS never leaves the host. Have your HTTP clients bypass web proxies within the VM when querying IMDS, and treat169.254.169.254the same as168.63.129.16. For more details, see Azure Instance Metadata Service. Thus, this IP should not be part of any routing and is meant to be accessed locally only. - Are you able to use the PowerShell cmd below to get the output?
This is similar to what the services/agent on the machine would be querying the IMDS service regularly. Check if the output returned is similar to what is mentioned hereInvoke-RestMethod -Headers @{"Metadata"="true"} -Method GET -NoProxy -Uri "http://169.254.169.254/metadata/instance?api-version=2021-02-01" | ConvertTo-Json -Depth 64 - How many machines are you facing this issue? I would suggest reviewing networking/proxy/security differences between the machines where the error is not available and where you get the error.
Also, note that The Log Analytics agent is on a deprecation path and won't be supported after August 31, 2024. If you use the Log Analytics agent to ingest data to Azure Monitor, migrate to the new Azure Monitor agent prior to that date.
Hope this helps. Looking forward to your response.
-
coolqwerty 61 Reputation points
2024-03-19T07:53:00.78+00:00 Hi Anurag ,
Yes i am seeing legacy agent only. This is working for other VM with same legacy agent. Also all the servers are using proxy so on previous case MS engineer suggested to put the route route add 169.254.169.254/32 <VM IP> metric -p .after adding this route to all VM they started working fine and reporting all values in heartbeat. But only this VM is not working as expected after adding the route. The reason MS suggested to add route to bypass the proxy because IMDS IP works with no proxy.
Also note that heartbeat is coming regularly only just that particular paramter Subscription ID which is a part of heartbeat when you expand it is coming missing.
what could be the cause of this ?
-
AnuragSingh-MSFT 21,566 Reputation points Moderator
2024-03-19T08:21:56.69+00:00 coolqwerty, thank you for the reply.
Could you try step 4 in my reply above from the local machine and check the information that you are getting - whether it matches the output as received from other "working" servers.
-
coolqwerty 61 Reputation points
2024-03-19T08:43:46.9333333+00:00 this the error we are getting
-
AnuragSingh-MSFT 21,566 Reputation points Moderator
2024-03-20T08:00:32.9666667+00:00 coolqwerty, thank you for the reply. It seems that you are using PowerShell 5.1 version.
If you have PowerShell core installed, please launch PScore shell and use the cmdlet. Else, with older version of PowerShell, the following can be used:
$Proxy=New-object System.Net.WebProxy $WebSession=new-object Microsoft.PowerShell.Commands.WebRequestSession $WebSession.Proxy=$Proxy Invoke-RestMethod -Headers @{"Metadata"="true"} -Method GET -Uri "http://169.254.169.254/metadata/instance?api-version=2021-02-01" -WebSession $WebSession | ConvertTo-Json -Depth 64For details, see https://github.com/microsoft/azureimds/blob/master/IMDSSample.ps1
Hope this helps.
-
coolqwerty 61 Reputation points
2024-03-20T09:29:12.4233333+00:00 Yes Anurag this above powershell worked this time and in output i can see that it is pulling out subscription ID and resource group fine but why it is not getting populated in the log analytics workspace when we are querying heartbeat data. Please note that when we query for heartbeat and expand each heartbeta then we see all data except subscription ID and resource grp .. these 2 values missing in heartbeat data
Please let me know where the issue could be
-
coolqwerty 61 Reputation points
2024-03-21T07:29:21.84+00:00 Is there anything else i can check ? From the command above i can see subscription ID and resourceID is coming in result output but not getting into Log analytics. In heartbeat output i still see Subscription ID is not coming.
-
AnuragSingh-MSFT 21,566 Reputation points Moderator
2024-03-22T03:25:28.3933333+00:00 coolqwerty, thank you for the reply. Based on the error in OperationsManager logs (400 bad request), the requests getting to IMDS are not properly formed (maybe it is still using proxy).
I would suggest clearing the HealthService cache of Microsoft Monitoring Agent, by following the steps below:
Navigate to the client machine.
- Stop the Microsoft Monitoring Agent (HealthService) service.
- Rename the cache folder:
%ProgramFiles%\Microsoft Monitoring Agent\Agent\Health Service Stateto%ProgramFiles%\Microsoft Monitoring Agent\Agent\Health Service State.old - Restart the Microsoft Monitoring Agent (HealthService) service.
Monitor the OperationsManager logs for the sequence of events coming up and note down any communication errors coming up. For more details, see How and When to Clear the Cache. Although this article applies to SCOM, the agent used is the same in both cases.
Hope this helps.
-
coolqwerty 61 Reputation points
2024-03-22T13:26:47.2166667+00:00 Hi Anurag ,
Thanks for the steps you provided. I performed those but still the issue is not resolved. Still missing those 2 values in heartbeat .
-
AnuragSingh-MSFT 21,566 Reputation points Moderator
2024-03-26T05:31:56.09+00:00 coolqwerty, thank you for the reply. Please check and answer the queries below:
- Can you check and confirm the set of errors that you are getting in OperationsManager event log (and its frequency) after you performed the steps above.
- Confirm if this agent is also getting monitored using SCOM, apart from being connected to LA workspace?
- Ensure that the OperationsManager agent is updated with the latest version.
- In workspace, check if you are getting the
ResourceIdfield populated for this VM in the heartbeat table using the query below:Heartbeat | where Computer has "<VMName>" | summarize arg_max(TimeGenerated, *) | project TimeGenerated, Computer, ResourceId
Please let me know if you have any questions.
-
coolqwerty 61 Reputation points
2024-03-26T08:32:27.4333333+00:00 I ran this query and it is giving Time generated ,Computer name but Resource ID is coming blank.
-
AnuragSingh-MSFT 21,566 Reputation points Moderator
2024-03-27T06:23:19.42+00:00 coolqwerty, thank you for the details. I would suggest performing the following operations:
- Can you check and confirm the set of errors that you are getting in OperationsManager event log and its frequency? This should give an idea of what the cause of error might be.
- Ensure that the OperationsManager agent is updated with the latest version.
If the steps don't narrow down /resolve the issue, I would suggest reaching out to Microsoft Support for further assistance. Based on the nature of the issue, detailed traces/logs from the server will have to be collected to understand the cause of the error.
-
coolqwerty 61 Reputation points
2024-03-27T08:10:17.97+00:00 Hi Anurag ,
Yes i have raised a support ticket with MS . I will post here the resolution i recieved from MS .
Thank You
-
AnuragSingh-MSFT 21,566 Reputation points Moderator
2024-04-04T09:40:25.91+00:00 coolqwerty, following up to check if troubleshooting has resulted in figuring out the root cause of this issue.
-
coolqwerty 61 Reputation points
2024-04-04T09:44:10.8733333+00:00 We opened MS support ticket but MS also was not able to figure out what was the issue.
finally after the server got patched and rebooted it is now reporting the missing subs ID and resource ID in the log analytics heartbeat data .
Sign in to comment
Sign in to answer