Hybrid Worker Extension on an Azure Arc-connected server stops working after a week.

Question

Hybrid Worker Extension on an Azure Arc-connected server stops working after a week.

Simon Enbom 10

I have a hybrid worker group setup on one Azure Arc-connected machine. The hybrid worker extension is installed, and initially, it works great.

However, after roughly one week, I get the error with Event ID 15105 in Event Viewer: - The task category is HybridErrorWhilePullingQueue

"Error while polling the hybrid notification channel <accountid, runbookworkergroup, machinename, etc>"

"}] [Exception=JobRuntimeData.Client.JobRuntimeDataServiceClientException: The remote server returned an error: (403) Forbidden."

In Azure Automation, the jobs get suspended. If I click into the jobs itself, the exception tab shows "Job was suspended as the job limit for a Hybrid Worker exceeded. Add more Hybrid Workers to the Hybrid Worker group to overcome this issue, see: https://aka.ms/HRW-polling-limit-exceed"

However, I only have one job. And it's not repeating, it's trigger based, and I can see that the trigger hasn't fired. The hybrid worker has a "last seen time" in Azure that's just a few minutes old.

Re-installing the extension WORKS. But it's been doing this for a few weeks now, where it will continute to work for about one week, before requiring a re-install to get up and running again. How can I prevent this? It's not feasible to have to re-install it every single week.

Running the troubleshooter, this is the only part that fails.

RuleId                      : HybridExtensionLogCheck
RuleGroupId                 : Extensionhealth
RuleName                    : Hybrid Extension log check
RuleGroupName               : Hybrid Extension log Checks
RuleDescription             : Analyze latest error event logs of hybrid extension
CheckResult                 : Failed
CheckResultMessage          : There are no hybrid extension logs, Extension installation may have some issue. Please re
                              install the extension.
CheckResultMessageId        : HybridExtensionLogCheck.Failed.There are no hybrid extension logs
CheckResultMessageArguments : {}

Izmir Sadigov 111 Reputation points MVP

2023-05-24T09:11:17.6+00:00

Ensure that the server has proper network connectivity to Azure services. Check if the server can reach the Azure Arc-enabled server endpoints and if any firewalls or network security groups are blocking the communication.

Make sure that the Hybrid Worker Extension is up to date. Check the Azure portal for any available updates and apply them if necessary.
Simon Enbom 10 Reputation points

2023-05-24T10:19:37.25+00:00

Thanks. As I said, it works fine, for a week, then it stops working. No configuration changes are made during this time.
Alexey Suslin 0 Reputation points

2025-03-18T16:53:41.43+00:00

<DELETED WRONG PLACE>
Alexey Suslin 0 Reputation points

2025-03-18T16:56:08.66+00:00

Are you able to understand the issue? we got the same like 1 month - everything was working and then suddenly stopped running all runbooks, I stopped everything and just try to run simple "Hello World" but the same error: "Job was suspended as the job limit for a Hybrid Worker exceeded. Add more Hybrid Workers to the Hybrid Worker group to overcome this issue"

1 answer

Your answer

Izmir Sadigov 111 Reputation points MVP

2023-05-24T09:11:17.6+00:00

Ensure that the server has proper network connectivity to Azure services. Check if the server can reach the Azure Arc-enabled server endpoints and if any firewalls or network security groups are blocking the communication.

Make sure that the Hybrid Worker Extension is up to date. Check the Azure portal for any available updates and apply them if necessary.
Simon Enbom 10 Reputation points

2023-05-24T10:19:37.25+00:00

Thanks. As I said, it works fine, for a week, then it stops working. No configuration changes are made during this time.
Alexey Suslin 0 Reputation points

2025-03-18T16:53:41.43+00:00

<DELETED WRONG PLACE>
Alexey Suslin 0 Reputation points

2025-03-18T16:56:08.66+00:00

Are you able to understand the issue? we got the same like 1 month - everything was working and then suddenly stopped running all runbooks, I stopped everything and just try to run simple "Hello World" but the same error: "Job was suspended as the job limit for a Hybrid Worker exceeded. Add more Hybrid Workers to the Hybrid Worker group to overcome this issue"

Answer 1

Simon Enbom 10

In the end in my case, it was completely un-related to Azure Arc and Azure Automation. Sort of... we had a DR tool that took backup of the machine where the worker extension was, and spun it up in a seperate environment for DR testing. Since I weren't aware of the testing, and there was no information about the actual machine (like IP, mac, or something) in Azure, it took some digging to find out.

When we found out why, it was easy to fix however.

Alexey Suslin 0 Reputation points

2025-03-20T08:16:40.4966667+00:00

my issue was solved because of the wrong resolution ip address of the automation account link. it was started resolving to public ip address instead of private. I was able to fix it by tuning DNS Server for hybrid worker. However the initial error message saying that I have to add more hybrid workers into the pool was misleading

Share via

Hybrid Worker Extension on an Azure Arc-connected server stops working after a week.

1 answer

Your answer