Heartbeat data missing some data .error -Microsoft.EnterpriseManagement.HealthService.HealthAssessmentData.HealthAssessmentDataSource" reported an exception System.Net.WebException: The remote server returned an error: (400) Bad Request

coolqwerty 21

Some of the heartbeat data is missing such as subscription ID . In the operation manager logs i am seeing below error which is the real cause of the problem - In below error is not able to connect with IMDS which has IP 169.254.169.254.

The establish the connectivity we have added the route so that VM IP can directly connect with 169.254.169.254 by bypassing the proxy but still getting 400 bad request.

Does anyone know solution of this ?

===============================================

Microsoft.EnterpriseManagement.HealthService.HealthAssessmentData.HealthAssessmentDataSource" reported an exception System.Net.WebException: The remote server returned an error: (400) Bad Request.

at System.Net.HttpWebRequest.GetResponse()

at Microsoft.EnterpriseManagement.HealthService.HealthAssessmentData.HealthAssessmentDataSource.TryGetIMDSData(Object state)

AnuragSingh-MSFT 21,251 Reputation points

2024-03-19T06:24:58.4266667+00:00
coolqwerty, thank you for posting this question on Microsoft Q&A. Could you please follow the steps below/answer the queries to help us narrow down the issue further:

You mentioned that you have seen this error in the "Operations Manager" logs. Are you using LA agent on this VM and not the AMA agent?

How frequently do you see this error? The heartbeats are triggered every 1 minute, if the frequency of this error varies, it could mean intermittent issue on the machine itself.

IMDS is a REST API that's available at a well-known, non-routable IP address (169.254.169.254). You can only access it from within the VM. Communication between the VM and IMDS never leaves the host. Have your HTTP clients bypass web proxies within the VM when querying IMDS, and treat 169.254.169.254 the same as 168.63.129.16. For more details, see Azure Instance Metadata Service. Thus, this IP should not be part of any routing and is meant to be accessed locally only.

Are you able to use the PowerShell cmd below to get the output?
Invoke-RestMethod -Headers @{"Metadata"="true"} -Method GET -NoProxy -Uri "http://169.254.169.254/metadata/instance?api-version=2021-02-01" | ConvertTo-Json -Depth 64
This is similar to what the services/agent on the machine would be querying the IMDS service regularly. Check if the output returned is similar to what is mentioned here

How many machines are you facing this issue? I would suggest reviewing networking/proxy/security differences between the machines where the error is not available and where you get the error.

Also, note that The Log Analytics agent is on a deprecation path and won't be supported after August 31, 2024. If you use the Log Analytics agent to ingest data to Azure Monitor, migrate to the new Azure Monitor agent prior to that date.

Hope this helps. Looking forward to your response.
coolqwerty 21 Reputation points

2024-03-19T07:53:00.78+00:00

Hi Anurag ,

Yes i am seeing legacy agent only. This is working for other VM with same legacy agent. Also all the servers are using proxy so on previous case MS engineer suggested to put the route route add 169.254.169.254/32 <VM IP> metric -p .after adding this route to all VM they started working fine and reporting all values in heartbeat. But only this VM is not working as expected after adding the route. The reason MS suggested to add route to bypass the proxy because IMDS IP works with no proxy.

Also note that heartbeat is coming regularly only just that particular paramter Subscription ID which is a part of heartbeat when you expand it is coming missing.

what could be the cause of this ?
AnuragSingh-MSFT 21,251 Reputation points

2024-03-19T08:21:56.69+00:00

coolqwerty, thank you for the reply.

Could you try step 4 in my reply above from the local machine and check the information that you are getting - whether it matches the output as received from other "working" servers.
coolqwerty 21 Reputation points

2024-03-19T08:43:46.9333333+00:00

this the error we are getting
AnuragSingh-MSFT 21,251 Reputation points

2024-03-20T08:00:32.9666667+00:00
coolqwerty, thank you for the reply. It seems that you are using PowerShell 5.1 version.

If you have PowerShell core installed, please launch PScore shell and use the cmdlet. Else, with older version of PowerShell, the following can be used:

$Proxy=New-object System.Net.WebProxy $WebSession=new-object Microsoft.PowerShell.Commands.WebRequestSession $WebSession.Proxy=$Proxy Invoke-RestMethod -Headers @{"Metadata"="true"} -Method GET -Uri "http://169.254.169.254/metadata/instance?api-version=2021-02-01" -WebSession $WebSession | ConvertTo-Json -Depth 64

For details, see https://github.com/microsoft/azureimds/blob/master/IMDSSample.ps1

Hope this helps.
coolqwerty 21 Reputation points

2024-03-20T09:29:12.4233333+00:00

Yes Anurag this above powershell worked this time and in output i can see that it is pulling out subscription ID and resource group fine but why it is not getting populated in the log analytics workspace when we are querying heartbeat data. Please note that when we query for heartbeat and expand each heartbeta then we see all data except subscription ID and resource grp .. these 2 values missing in heartbeat data

Please let me know where the issue could be
coolqwerty 21 Reputation points

2024-03-21T07:29:21.84+00:00

Is there anything else i can check ? From the command above i can see subscription ID and resourceID is coming in result output but not getting into Log analytics. In heartbeat output i still see Subscription ID is not coming.
AnuragSingh-MSFT 21,251 Reputation points

2024-03-22T03:25:28.3933333+00:00
coolqwerty, thank you for the reply. Based on the error in OperationsManager logs (400 bad request), the requests getting to IMDS are not properly formed (maybe it is still using proxy).

I would suggest clearing the HealthService cache of Microsoft Monitoring Agent, by following the steps below:

Navigate to the client machine.

Stop the Microsoft Monitoring Agent (HealthService) service.

Rename the cache folder: %ProgramFiles%\Microsoft Monitoring Agent\Agent\Health Service State to %ProgramFiles%\Microsoft Monitoring Agent\Agent\Health Service State.old

Restart the Microsoft Monitoring Agent (HealthService) service.

Monitor the OperationsManager logs for the sequence of events coming up and note down any communication errors coming up. For more details, see How and When to Clear the Cache. Although this article applies to SCOM, the agent used is the same in both cases.

Hope this helps.
coolqwerty 21 Reputation points

2024-03-22T13:26:47.2166667+00:00

Hi Anurag ,

Thanks for the steps you provided. I performed those but still the issue is not resolved. Still missing those 2 values in heartbeat .
AnuragSingh-MSFT 21,251 Reputation points

2024-03-26T05:31:56.09+00:00
coolqwerty, thank you for the reply. Please check and answer the queries below:

Can you check and confirm the set of errors that you are getting in OperationsManager event log (and its frequency) after you performed the steps above.

Confirm if this agent is also getting monitored using SCOM, apart from being connected to LA workspace?

Ensure that the OperationsManager agent is updated with the latest version.

In workspace, check if you are getting the ResourceId field populated for this VM in the heartbeat table using the query below:
Heartbeat | where Computer has "<VMName>" | summarize arg_max(TimeGenerated, *) | project TimeGenerated, Computer, ResourceId

Please let me know if you have any questions.
coolqwerty 21 Reputation points

2024-03-26T08:32:27.4333333+00:00

I ran this query and it is giving Time generated ,Computer name but Resource ID is coming blank.
AnuragSingh-MSFT 21,251 Reputation points

2024-03-27T06:23:19.42+00:00
coolqwerty, thank you for the details. I would suggest performing the following operations:

Can you check and confirm the set of errors that you are getting in OperationsManager event log and its frequency? This should give an idea of what the cause of error might be.

Ensure that the OperationsManager agent is updated with the latest version.

If the steps don't narrow down /resolve the issue, I would suggest reaching out to Microsoft Support for further assistance. Based on the nature of the issue, detailed traces/logs from the server will have to be collected to understand the cause of the error.
coolqwerty 21 Reputation points

2024-03-27T08:10:17.97+00:00

Hi Anurag ,

Yes i have raised a support ticket with MS . I will post here the resolution i recieved from MS .

Thank You
AnuragSingh-MSFT 21,251 Reputation points

2024-04-04T09:40:25.91+00:00

coolqwerty, following up to check if troubleshooting has resulted in figuring out the root cause of this issue.
coolqwerty 21 Reputation points

2024-04-04T09:44:10.8733333+00:00

We opened MS support ticket but MS also was not able to figure out what was the issue.

finally after the server got patched and rebooted it is now reporting the missing subs ID and resource ID in the log analytics heartbeat data .

Share via

Heartbeat data missing some data .error -Microsoft.EnterpriseManagement.HealthService.HealthAssessmentData.HealthAssessmentDataSource" reported an exception System.Net.WebException: The remote server returned an error: (400) Bad Request

Does anyone know solution of this ?