Timeout Running BVTs using Service Fabric

Question

Timeout Running BVTs using Service Fabric

Allison 20 Microsoft Intern

Hi,

I'm currently trying to run my BVTs using Service Fabric, but I'm running into timeout issues. When first running the runbvtd command on Command Prompt, all my applications successfully come up on Service Fabric. However, when running the tests, I run into timeout issues and some of my applications become unhealthy. It looks like the timeout is related to AzCiM and AzPEAdaptor.

User's image

I'm running these BVTs on a Dev Box. It is also interesting that the tests run successfully as a part of my PR on DevOps, so it looks like it's specific to the machine I'm using. However, I don't know why there would be isolated issues for me if I'm using the same Dev Box as the rest of my team who can run them successfully.

Any help would be greatly appreciated.

Thank you!

Arko 4,150 Reputation points Microsoft External Staff Moderator

2025-06-17T06:38:12.8+00:00

Hello Allison,

From what you've described and based on the logs it looks like the root cause of the timeout is related to the tenant creation process failing to complete within the expected time in AzPEAdaptor. Specifically, the line Tenant creation in AzPEAdaptor did not happen in 900 attempts. Throwing along with the repeated container polling messages Container state Started is either not Started, or goal state has not been achieved clearly shows that the test infrastructure is waiting for containers to fully start, but the goal state is never reached. As a result, your test Test_Hibernation_Workflow_For_MultiInstance_Tenant is timing out.

You also mentioned that the same BVTs work fine when run through your DevOps PR pipeline, which confirms that the issue is isolated to your Dev Box. Please check if you have already tried the below suggested steps.

Restart your Dev Box to ensure you're starting from a clean state. Sometimes leftover containers or processes can interfere with goal state initialization. Run runbvtd -clean before testing again, just to make sure there's no stale state or partially created tenant hanging around. Validate Docker/Moby engine health on your box. If the container runtime itself is misconfigured or overloaded, it can cause the container state to hang. Running docker info can help spot any issues.

As per my understanding, the tests are failing locally because the container services that your test depends on (like AzCiM and ExhibitSinkService) aren't coming up quickly enough, which leads to the goal state not being achieved within the allowed time. Since these same tests pass in your CI pipeline, it's very likely a Dev Box–specific issue rather than a problem in the test itself.

Let me know your findings, thanks.
Arko 4,150 Reputation points Microsoft External Staff Moderator

2025-06-18T11:08:41.9266667+00:00

Hello Allison, Following up on this. Hope I was able to clear your query. Thanks
Allison 20 Reputation points Microsoft Intern

2025-06-19T17:20:47.1266667+00:00

Hi, I'm still running into issues. I'm planning on seeing if creating a new Dev Box helps.
Allison 20 Reputation points Microsoft Intern

2025-06-19T20:30:25.0566667+00:00

It's still not working.
Arko 4,150 Reputation points Microsoft External Staff Moderator

2025-06-20T08:24:31.3833333+00:00

Is it still not working after creating a dev box? What is the latest error are you getting? Please do a runbvtd -clean right away to clear any default artifacts and then try and let me know if you are seeing the same issue? If yes then share with me the latest error message and logs. I will take a look at it.
Allison 20 Reputation points Microsoft Intern

2025-06-20T16:52:30+00:00

I'm still running into the exact same issue, "Test 'Test_GoalState_Workflow_For_SingleInstance_Tenant' exceeded execution timeout period."
Arko 4,150 Reputation points Microsoft External Staff Moderator

2025-06-20T18:31:15.1566667+00:00
Are you behind the AZVPN on your Dev Box? I guess you are trying this from your corporate laptop and it is connected to a vpn. In some setups, outbound connectivity to required services e.g. AzPE backend can silently fail or time out.

Force restart the AzPEAdaptor container or if you have direct control or try stopping all containers and rerunning

docker stop $(docker ps -q) runbvtd -clean runbvtd

If the container consistently hangs at the AzPEAdaptor stage across clean boxes, then there’s a chance this might be a broader backend timing issue. Let me know what you find after checking those points, and I’ll be happy to help further based on the output.

Your answer

Arko 4,150 Reputation points Microsoft External Staff Moderator

2025-06-17T06:38:12.8+00:00

Hello Allison,

From what you've described and based on the logs it looks like the root cause of the timeout is related to the tenant creation process failing to complete within the expected time in AzPEAdaptor. Specifically, the line Tenant creation in AzPEAdaptor did not happen in 900 attempts. Throwing along with the repeated container polling messages Container state Started is either not Started, or goal state has not been achieved clearly shows that the test infrastructure is waiting for containers to fully start, but the goal state is never reached. As a result, your test Test_Hibernation_Workflow_For_MultiInstance_Tenant is timing out.

You also mentioned that the same BVTs work fine when run through your DevOps PR pipeline, which confirms that the issue is isolated to your Dev Box. Please check if you have already tried the below suggested steps.

Restart your Dev Box to ensure you're starting from a clean state. Sometimes leftover containers or processes can interfere with goal state initialization. Run runbvtd -clean before testing again, just to make sure there's no stale state or partially created tenant hanging around. Validate Docker/Moby engine health on your box. If the container runtime itself is misconfigured or overloaded, it can cause the container state to hang. Running docker info can help spot any issues.

As per my understanding, the tests are failing locally because the container services that your test depends on (like AzCiM and ExhibitSinkService) aren't coming up quickly enough, which leads to the goal state not being achieved within the allowed time. Since these same tests pass in your CI pipeline, it's very likely a Dev Box–specific issue rather than a problem in the test itself.

Let me know your findings, thanks.
Arko 4,150 Reputation points Microsoft External Staff Moderator

2025-06-18T11:08:41.9266667+00:00

Hello Allison, Following up on this. Hope I was able to clear your query. Thanks
Allison 20 Reputation points Microsoft Intern

2025-06-19T17:20:47.1266667+00:00

Hi, I'm still running into issues. I'm planning on seeing if creating a new Dev Box helps.
Allison 20 Reputation points Microsoft Intern

2025-06-19T20:30:25.0566667+00:00

It's still not working.
Arko 4,150 Reputation points Microsoft External Staff Moderator

2025-06-20T08:24:31.3833333+00:00

Is it still not working after creating a dev box? What is the latest error are you getting? Please do a runbvtd -clean right away to clear any default artifacts and then try and let me know if you are seeing the same issue? If yes then share with me the latest error message and logs. I will take a look at it.
Allison 20 Reputation points Microsoft Intern

2025-06-20T16:52:30+00:00

I'm still running into the exact same issue, "Test 'Test_GoalState_Workflow_For_SingleInstance_Tenant' exceeded execution timeout period."
Arko 4,150 Reputation points Microsoft External Staff Moderator

2025-06-20T18:31:15.1566667+00:00

Are you behind the AZVPN on your Dev Box? I guess you are trying this from your corporate laptop and it is connected to a vpn. In some setups, outbound connectivity to required services e.g. AzPE backend can silently fail or time out.

Force restart the AzPEAdaptor container or if you have direct control or try stopping all containers and rerunning

docker stop $(docker ps -q) runbvtd -clean runbvtd

If the container consistently hangs at the AzPEAdaptor stage across clean boxes, then there’s a chance this might be a broader backend timing issue. Let me know what you find after checking those points, and I’ll be happy to help further based on the output.

Share via

Timeout Running BVTs using Service Fabric

Your answer