##[error]We stopped hearing from agent Azure Pipelines 4. Verify the agent machine is running and has a healthy network connection. Anything that terminates an agent process, starves it for CPU, or blocks its network access can cause this error..

Question

##[error]We stopped hearing from agent Azure Pipelines 4. Verify the agent machine is running and has a healthy network connection. Anything that terminates an agent process, starves it for CPU, or blocks its network access can cause this error..

Ranjan, Rajeev (KT) 50

##[error]We stopped hearing from agent Azure Pipelines 4. Verify the agent machine is running and 
	  has a healthy network connection. 
	  Anything that terminates an agent process, starves it for CPU, or blocks its network access can cause this error. 
	  For more information, see: https://go.microsoft.com/fwlink/?linkid=846610

      This job was abandoned. We have detected that logs from the agent may have not finished uploading. 
	  We have included our in-memory record of all log lines uploaded before we lost contact with the agent.

Aaron Grove 0 Reputation points

2023-05-15T16:28:03.99+00:00

I have the same issue, the release works fine for deploying to dev and test, but fails when deploying to uat. Takes about 7 mins for dev/test, but gets to over 1hr before failing for uat
Kolbe, Florian 0 Reputation points

2023-05-17T06:30:06.2433333+00:00

Same here on ubuntu. Last logged step (last thing running?) was an angular cli build

This job was abandoned. We have detected that logs from the agent may have not finished uploading. We have included our in-memory record of all log lines uploaded before we lost contact with the agent:
sireesha madireddy 0 Reputation points

2023-07-25T03:01:27.34+00:00

Hi @Ranjan Rajeev , Did you managed to resolve this issue? Am also getting the same error message and build is failing. Please suggest if you found any solution for this issue.
Enda Brody 0 Reputation points

2024-02-09T16:47:28.8833333+00:00

Hi I am getting this error when running Selenium maven tests. I tried breaking them up but still no joy the tests do have to run as they are. ##[error]We stopped hearing from agent Hosted Agent. Verify the agent machine is running and has a healthy network connection. Anything that terminates an agent process, starves it for CPU, or blocks its network access can cause this error. For more information, see: https://go.microsoft.com/fwlink/?linkid=846610
Deng, Lea 0 Reputation points

2024-05-24T01:58:23.69+00:00

I had the same issue multiple times this week with self hosted agent version 3.239.1, Agent.OS is Windows_NT.

This happened when using DevOps pipelines and self-hosted agents to run ReadyAPI tests in parallel

The agent has accepted the job request, but the job got stuck at "initialize job" stage. After about 13 minutes, the pipeline failed with error message below:

"##[error]We stopped hearing from agent agentTFSServer4. Verify the agent machine is running and has a healthy network connection. Anything that terminates an agent process, starves it for CPU, or blocks its network access can cause this error. For more information, see: https://go.microsoft.com/fwlink/?linkid=846610"

Looking at the agent log, we can see MessageListener repeated sending GetAgentMessage for about 11 minutes. Eventually we got visual studio error message:

"PATCH request to https://dev.azure.com/cccgovtnz/_apis/distributedtask/pools/35/jobrequests/19344 failed. HTTP Status: BadRequest, "

full logging below:

[2024-05-20 03:19:21Z INFO JobDispatcher] Job request 19344 for plan d75273f5-3cc3-42c7-9095-dcc1f355a53a job 275f1d19-1bd8-5591-b06b-07d489ea915a received.

[2024-05-20 03:19:21Z INFO Terminal] WRITE LINE: 2024-05-20 03:19:21Z: Running job: Agent job - parallel readyAPI Test Use Repo3

[2024-05-20 03:19:52Z INFO MessageListener] Sent GetAgentMessage to keep alive agent 80, session '89294bff-b3eb-417a-afb0-2f951587c456'.

[2024-05-20 03:20:23Z INFO MessageListener] Sent GetAgentMessage to keep alive agent 80, session '89294bff-b3eb-417a-afb0-2f951587c456'.

[2024-05-20 03:20:54Z INFO MessageListener] Sent GetAgentMessage to keep alive agent 80, session '89294bff-b3eb-417a-afb0-2f951587c456'.

[2024-05-20 03:21:25Z INFO MessageListener] Sent GetAgentMessage to keep alive agent 80, session '89294bff-b3eb-417a-afb0-2f951587c456'.

[2024-05-20 03:21:56Z INFO MessageListener] Sent GetAgentMessage to keep alive agent 80, session '89294bff-b3eb-417a-afb0-2f951587c456'.

[2024-05-20 03:22:27Z INFO MessageListener] Sent GetAgentMessage to keep alive agent 80, session '89294bff-b3eb-417a-afb0-2f951587c456'.

[2024-05-20 03:22:58Z INFO MessageListener] Sent GetAgentMessage to keep alive agent 80, session '89294bff-b3eb-417a-afb0-2f951587c456'.

[2024-05-20 03:23:29Z INFO MessageListener] Sent GetAgentMessage to keep alive agent 80, session '89294bff-b3eb-417a-afb0-2f951587c456'.

[2024-05-20 03:24:00Z INFO MessageListener] Sent GetAgentMessage to keep alive agent 80, session '89294bff-b3eb-417a-afb0-2f951587c456'.

[2024-05-20 03:24:31Z INFO MessageListener] Sent GetAgentMessage to keep alive agent 80, session '89294bff-b3eb-417a-afb0-2f951587c456'.

[2024-05-20 03:25:02Z INFO MessageListener] Sent GetAgentMessage to keep alive agent 80, session '89294bff-b3eb-417a-afb0-2f951587c456'.

[2024-05-20 03:25:33Z INFO MessageListener] Sent GetAgentMessage to keep alive agent 80, session '89294bff-b3eb-417a-afb0-2f951587c456'.

[2024-05-20 03:26:04Z INFO MessageListener] Sent GetAgentMessage to keep alive agent 80, session '89294bff-b3eb-417a-afb0-2f951587c456'.

[2024-05-20 03:26:35Z INFO MessageListener] Sent GetAgentMessage to keep alive agent 80, session '89294bff-b3eb-417a-afb0-2f951587c456'.

[2024-05-20 03:27:06Z INFO MessageListener] Sent GetAgentMessage to keep alive agent 80, session '89294bff-b3eb-417a-afb0-2f951587c456'.

[2024-05-20 03:27:37Z INFO MessageListener] Sent GetAgentMessage to keep alive agent 80, session '89294bff-b3eb-417a-afb0-2f951587c456'.

[2024-05-20 03:28:09Z INFO MessageListener] Sent GetAgentMessage to keep alive agent 80, session '89294bff-b3eb-417a-afb0-2f951587c456'.

[2024-05-20 03:28:40Z INFO MessageListener] Sent GetAgentMessage to keep alive agent 80, session '89294bff-b3eb-417a-afb0-2f951587c456'.

[2024-05-20 03:29:11Z INFO MessageListener] Sent GetAgentMessage to keep alive agent 80, session '89294bff-b3eb-417a-afb0-2f951587c456'.

[2024-05-20 03:29:42Z INFO MessageListener] Sent GetAgentMessage to keep alive agent 80, session '89294bff-b3eb-417a-afb0-2f951587c456'.

[2024-05-20 03:30:13Z INFO MessageListener] Sent GetAgentMessage to keep alive agent 80, session '89294bff-b3eb-417a-afb0-2f951587c456'.

[2024-05-20 03:30:39Z INFO AgentServer] Refresh JobRequest VssConnection to get on a different AFD node.

[2024-05-20 03:30:39Z INFO AgentServer] Establish connection with 30 seconds timeout.

[2024-05-20 03:30:39Z INFO VisualStudioServices] Starting operation Location.GetConnectionData

[2024-05-20 03:30:39Z INFO VisualStudioServices] Finished operation Location.GetConnectionData

[2024-05-20 03:30:39Z INFO JobDispatcher] Start renew job request 19344 for job 275f1d19-1bd8-5591-b06b-07d489ea915a.

[2024-05-20 03:30:39Z ERR VisualStudioServices] PATCH request to https://dev.azure.com/cccgovtnz/_apis/distributedtask/pools/35/jobrequests/19344 failed. HTTP Status: BadRequest, AFD Ref: Ref A: F32BF2EFB8EF4E7A9CD5388CA79CD0B7 Ref B: AKL30EDGE0210 Ref C: 2024-05-20T03:30:39Z

[2024-05-20 03:30:39Z INFO JobDispatcher] TaskAgentJobTokenExpiredException received renew job request 19344, job is no longer valid, stop renew job request.

[2024-05-20 03:30:39Z INFO JobDispatcher] Unable to renew job request for job 275f1d19-1bd8-5591-b06b-07d489ea915a for the first time, stop dispatching job to worker.
Chris Delaforce 0 Reputation points

2025-04-25T15:37:15.9133333+00:00

Are you guys using the Free ADO tier? There's a 60 min hard limit on job duration for MS Hosted agents at the free tier. This cannot be changed in the yaml, you have to pay for at least 1 extra parallel job tier https://learn.microsoft.com/en-us/azure/devops/pipelines/process/phases?view=azure-devops&tabs=yaml see Timeouts section.
So, you can either split your 60min+ job into 2 jobs and run them consecutively using dependencies between jobs and run two <60min jobs. Or if you cant do that you'll have to pay the £30/m for the extra capacity.
Deng, Lea 0 Reputation points

2025-04-28T03:24:21.4966667+00:00

Hi @Chris Delaforce I use self hosted agent and there's no duration limit.

2 answers

Your answer

Aaron Grove 0 Reputation points

2023-05-15T16:28:03.99+00:00

I have the same issue, the release works fine for deploying to dev and test, but fails when deploying to uat. Takes about 7 mins for dev/test, but gets to over 1hr before failing for uat
Kolbe, Florian 0 Reputation points

2023-05-17T06:30:06.2433333+00:00

Same here on ubuntu. Last logged step (last thing running?) was an angular cli build

This job was abandoned. We have detected that logs from the agent may have not finished uploading. We have included our in-memory record of all log lines uploaded before we lost contact with the agent:
sireesha madireddy 0 Reputation points

2023-07-25T03:01:27.34+00:00

Hi @Ranjan Rajeev , Did you managed to resolve this issue? Am also getting the same error message and build is failing. Please suggest if you found any solution for this issue.
Enda Brody 0 Reputation points

2024-02-09T16:47:28.8833333+00:00

Hi I am getting this error when running Selenium maven tests. I tried breaking them up but still no joy the tests do have to run as they are. ##[error]We stopped hearing from agent Hosted Agent. Verify the agent machine is running and has a healthy network connection. Anything that terminates an agent process, starves it for CPU, or blocks its network access can cause this error. For more information, see: https://go.microsoft.com/fwlink/?linkid=846610
Chris Delaforce 0 Reputation points

2025-04-25T15:37:15.9133333+00:00

Are you guys using the Free ADO tier? There's a 60 min hard limit on job duration for MS Hosted agents at the free tier. This cannot be changed in the yaml, you have to pay for at least 1 extra parallel job tier https://learn.microsoft.com/en-us/azure/devops/pipelines/process/phases?view=azure-devops&tabs=yaml see Timeouts section.
So, you can either split your 60min+ job into 2 jobs and run them consecutively using dependencies between jobs and run two <60min jobs. Or if you cant do that you'll have to pay the £30/m for the extra capacity.
Deng, Lea 0 Reputation points

2025-04-28T03:24:21.4966667+00:00

Hi @Chris Delaforce I use self hosted agent and there's no duration limit.

Answer 1

Jayesh Patel 10

Hi guys, im getting the same issue using a hotst agent, tried windows 2022 and windows latest same result. the pipeline is only running for 1hrs, i can see the job is actual complete but it fails with this error:

This job was abandoned. We have detected that logs from the agent may have not finished uploading. We have included our in-memory record of all log lines uploaded before we lost contact with the agent:

Jatinderpal Singh 0 Reputation points

2023-04-18T10:00:02.8633333+00:00

I am also facing the same issue with self hosted agent. Job is running for around 1hr 20 minutes and then pipeline fails with the said error. Any help would be appreciated Thank you!
Jayesh Patel 10 Reputation points

2023-06-21T14:03:03.9133333+00:00

Anyone solved this yet still facing issue
Mayur Malekar 0 Reputation points

2024-05-02T04:40:43.6566667+00:00

was anyone able to solve this
Diego Rafael Souza 0 Reputation points

2024-11-14T19:07:38.55+00:00

This is a comment, not an answer. Still, same issue here.

Answer 2

I had the same issue few months ago while using Linux self hosted agents. For that particular case, it turned up that there was a OS firmware update process running for my agent host that was killing the azdo agent process, as you can see in the journal logs:

fwupd[2844]: 18:47:15.266 FuMain               Received SIGTERM
systemd[1]: Stopping fwupd.service - Firmware update daemon...
systemd[1]: Stopping multipathd.service - Device-Mapper Multipath Device Controller...
multipathd[198]: multipathd: shut down
systemd[1]: Stopping packagekit.service - PackageKit Daemon...
python3[904]: 2024-11-14T18:47:15.271871Z INFO Daemon Daemon Agent WALinuxAgent-2.9.1.1 forwarding signal 15 to WALinuxAgent-2.12.0.2
systemd[1]: Stopping walinuxagent.service - Azure Linux Agent...
systemd[1]: multipathd.service: Deactivated successfully.
systemd[1]: Stopped multipathd.service - Device-Mapper Multipath Device Controller.
systemd[1]: packagekit.service: Deactivated successfully.
systemd[1]: Stopped packagekit.service - PackageKit Daemon.
runuser[2772]: pam_unix(runuser:session): session closed for user AzDevOps
sudo[2769]: pam_unix(sudo:session): session closed for user root
systemd[1]: Starting multipathd.service - Device-Mapper Multipath Device Controller...
systemd[1]: Starting packagekit.service - PackageKit Daemon...
systemd[1]: fwupd.service: Deactivated successfully.
systemd[1]: Stopped fwupd.service - Firmware update daemon.
systemd[1]: fwupd.service: Consumed 1.013s CPU time, 143.6M memory peak, 0B memory swap peak.
PackageKit[5707]: daemon start
systemd[1]: Starting fwupd.service - Firmware update daemon...
multipathd[5710]: multipathd v0.9.4: start up
multipathd[5710]: reconfigure: setting up paths and maps

What solved the issue for us was turn off the service that performs that OS update and was triggering the SIGTERM for the AZDO agent:

sudo systemctl disable apt-daily.timer
sudo systemctl disable apt-daily-upgrade.timer
sudo systemctl stop apt-daily.timer
sudo systemctl stop apt-daily-upgrade.timer

You can find more info at the github issue related: https://github.com/microsoft/azure-pipelines-agent/issues/4313#issuecomment-2501564255

Share via

##[error]We stopped hearing from agent Azure Pipelines 4. Verify the agent machine is running and has a healthy network connection. Anything that terminates an agent process, starves it for CPU, or blocks its network access can cause this error..

2 answers

Your answer