Azure Automation
An Azure service that is used to automate, configure, and install updates across hybrid environments.
1,113 questions
jobs are not able to run on Hybrid worker
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(57.599999999999994, 4%, 15%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAT%3C/text%3E%3C/svg%3E)
Ava Tomlinson
96
Reputation points
We have are using Hybrid Worker(Windows Server 2016) in an Azure Automation Account to run the PowerShell runbooks. We have noticed that the jobs in the Automation account are getting queued and suspended we have thousands of suspended jobs within last 24hours. The jobs are not able to run on Hybrid worker. We have checked the Microsoft SMA logs and found below error.
Event Viewer : Application and Services log- Operations Manager
- A module of type "Microsoft.EnterpriseManagement.HealthService.HealthAssessmentData.HealthAssessmentDataSource" reported an exception System.Net.WebException: The remote server returned an error: (504) Gateway Timeout.
at System.Net.HttpWebRequest.GetResponse()
at Microsoft.EnterpriseManagement.HealthService.HealthAssessmentData.HealthAssessmentDataSource.TryGetIMDSData(Object state) which was running as part of rule "CollectHealthAssessmentInformation" running for instance "" with id:"{XXXXXXXXXXXX}" in management group "XXXXXXXXXXXX". - A secure connection could not be negotiated with the service XXXXXXXXXXXX.ods.opinsights.azure.com. The article KB3126513 has additional troubleshooting information for connectivity issues. Possible reasons for this include:
The certificate authority present in server certificate is not in the "Third-Party Root Certification Authorities" store. Please add this authority to that store.
TLS 1.0, 1.1, and 1.2 are all disabled.
A suitable cypher suite could not be negotiated.
Other details:
Failure Code: 12175L
Any help is appreciated in advance. Thanks
{count} votes
-
Yi Lu_MSFT 17,456 Reputation points2022-04-19T02:04:04.143+00:00 Hi @Ava Tomlinson
This is a quick note to let you know that we are performing research on this issue. -
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(275.2, 41%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAM%3C/text%3E%3C/svg%3E)
AnuragSingh-MSFT 19,856 Reputation points2022-04-19T10:20:18.01+00:00 @Ava Tomlinson , @Ava Tomlinson , the screenshot above shows that LogAnalytics agent can connect to the workspace. Also, the error mentioned is related to one of the "workflow" to collect health of machine - not related to the HybridWorker's runbook execution. The following should help:
a. The
Application logsunderWindows Logs. This would contain logs related to installation failures/crashes happening for related processes.b. The
Operations Managerlogs under theApplications and Services Logswould contain errors related to connection to Log Analytics workspace.c. The
Operational logsavailable underApplications and Services Logs --> Microsoft-SMAin Event Viewer on the Hybrid worker machine. This would contain logs related to the functioning of runbook workers as well as execution of runbook jobs.d. Are the jobs failing for a particular runbook or set of runbooks?
e. Are all the "Hybrid Workers" online and running? Try to run a sample runbook using
option -> select the Hybrid Worker group, does it fail as well? f. The troubleshooting guides:
Extension-based
Agent-basedThese questions should help narrow down the issue further. Please let me know if you have any questions.
-
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(32, 1%, 13%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ECM%3C/text%3E%3C/svg%3E)
CaseyYang-MSFT 10,321 Reputation points2022-04-20T07:02:50.957+00:00 Hi @Ava Tomlinson ,
I removed the tag "office-sharepoint-server-administration" because this issue has little to do with it, Thank you for your understanding.
-
Ava Tomlinson 96 Reputation points
2022-04-20T07:18:11.97+00:00 a. No Error and critical log found in Application logs under Windows Logs.
b. Repetitive Error log in Operations Manager logs-
A module of type "Microsoft.EnterpriseManagement.HealthService.HealthAssessmentData.HealthAssessmentDataSource" reported an exception System.Net.WebException: The remote server returned an error: (504) Gateway Timeout.
at System.Net.HttpWebRequest.GetResponse()
at Microsoft.EnterpriseManagement.HealthService.HealthAssessmentData.HealthAssessmentDataSource.TryGetIMDSData(Object state) which was running as part of rule "CollectHealthAssessmentInformation" running for instance "" with id:"{8E33DF4F-C46F-F0B2-820E-EDDEC57520BF}" in management group "AOI-f7203428-76d2-4905-bf39-d7c21298d5e8".c. No Error, critical and warning log found in Microsoft-SMA
d. Jobs are getting Suspended for all the runbooks
e. There is only one Agent based hybrid worker and it is online. The jobs are getting queued for about 15minutes and getting Suspended.
(We cannot run any job manually as it is Production)f. We need some time to go through the troubleshooting articles shared by you.
Under services, Microsoft Monitoring agent is running. We have observed that “Microsoft Monitoring Agent APM”, “Microsoft Agent Auditing Forwarding” are in disable state where we are not sure whether these are relevant to this issue or not.
-
AnuragSingh-MSFT 19,856 Reputation points
2022-04-21T10:56:35.227+00:00 @Ava Tomlinson , thank you for the reply. The "Microsoft Monitoring Agent APM" is for application performance monitoring (.NET Framework based). This is not related to SMA/Azure Automation Hybrid runbook worker.
I understand that you are going through the troubleshooting guide as of now. Please ensure that you are following the guide depending on the type of Hybrid Worker (extension based OR agent based).
Please let me know with your findings and any questions that you may have.
-
AnuragSingh-MSFT 19,856 Reputation points
2022-04-25T11:08:59.857+00:00 @Ava Tomlinson , I wanted to check if you had a chance to follow the troubleshooting guide and investigate.
Please let me know if you have any questions.
Sign in to comment