Azure Storage Disks Disconnected from Virtual Machine When Running Test-Cluster Command

Question

Azure Storage Disks Disconnected from Virtual Machine When Running Test-Cluster Command

Jack 0

Description:

I am experiencing unexpected behaviour when running the Test-Cluster command in an Azure-based Windows Server 2016 environment. When executing the command, I observed the following issue:

Error Event 157: "Disk X has been surprise removed."

The affected disk automatically reconnected after the test completed.

This behaviour did not occur on all nodes—only 2 out of 3 servers had disks disconnect.

Environment Details:

OS: Windows Server 2016
Disks: Azure Managed Disks (not added to Failover Cluster)
Cluster Size: 3-node cluster
Previous Behaviour: The Test-Cluster command has been executed in the same environment before without this issue.
Comparison: I have also run Test-Cluster in an on-premise setup and in AWS environments, but this behaviour was not observed.

Questions:

Is this an expected behaviour when running Test-Cluster in Azure?
Has Microsoft documented any known issues regarding this occurrence? I had found this post - https://learn.microsoft.com/en-us/troubleshoot/azure/virtual-machines/windows/disk-has-been-surprise-removed-event-id-157?tabs=cluster but it doesn't directly related to the 'Test-Cluster' command
Is there an alternative method to validate the cluster's health that does not trigger disk disconnections?

Any guidance or official documentation reference on this issue would be greatly appreciated.

Anonymous

2025-03-12T15:12:09.51+00:00

Hi Jack,

Just checking in to see if the below answer provided by Marcin Policht helped.

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.

Thank you.

2 answers

Your answer

Anonymous

2025-03-12T15:12:09.51+00:00

Hi Jack,

Just checking in to see if the below answer provided by Marcin Policht helped.

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.

Thank you.

Answer 1

Marcin Policht 50,570 MVP Volunteer Moderator

Refer to https://learn.microsoft.com/en-us/troubleshoot/windows-server/high-availability/validate-hardware-failover-cluster#considerations-when-you-include-storage-tests

Considerations when you include storage tests

The Validate a Configuration Wizard runs all storage tests by default. All or some of the storage tests can be unselected by choosing the Run only tests I select option on the Testing Options page of the wizard. When storage tests are included, the Review Storage Status page of the wizard shows all of the disks and storage pools in the cluster and allows you to select the disks and storage pools to include in the storage tests. Storage tests require that a disk or storage pool that is assigned to a clustered role or Cluster Shared Volume be taken offline first. Therefore, anything using the storage won't have access to it during the storage tests. We recommend that any clustered role or other process that might be using the disk or storage pool is taken offline before the storage is included in the storage validation tests.

The Test-Cluster Windows PowerShell cmdlet runs all storage tests by default. You can specify the -Include parameter to run only storage tests or a specific storage test. You can use the -Disk and -Pool parameters to enable targeted storage validation. The -Disk parameter or the -Pool parameter allows specifying, respectively, one or more disks or storage pools to be included in the storage validation testing. If the -Disk parameter or the -Pool parameter is used to specify a disk or storage pool that is currently online and is assigned to a clustered role or Cluster Shared Volume, you must also specify the -Force parameter to validate the corresponding disk or storage pool; otherwise, you must ensure that the clustered disk or storage pool is offline before running the tests. If the -Disk parameter or the -Pool parameter isn't specified, Test-Cluster runs storage tests on all disks and storage pools that are available for cluster use or that are in the cluster resource offline or failed state. We recommend that any clustered role or other process that might be using the disk or storage pool is taken offline before the storage is included in the validation testing.

If the above response helps answer your question, remember to "Accept Answer" so that others in the community facing similar issues can easily find the solution. Your contribution is highly appreciated.

hth

Marcin

Jack 0 Reputation points

2025-03-12T16:10:32.6333333+00:00

Hi Marcin,

Thank you for your response. The confusion that I'm having is that we aren't using Clustered Storage (Cluster Shared Volumes), just standalone managed disks on each node. I can understand how there could be an impact if we were using clustered storage but the only role that is running on our Failover Cluster is SQL Always on Availability Groups.

Is it an expected behaviour that running 'Test-Cluster' would disconnect the storage drives on Azure VMs with attached Managed Disks?

Kind regards,

Jack
Jack 0 Reputation points

2025-03-12T16:11:59.06+00:00

edit - Duplicate comment - please ignore
Anonymous

2025-03-13T18:25:04.8233333+00:00
Hi Jack,

We have noticed that you have rated an answer as not helpful. We appreciate your feedback and are committed to improving your experience with the Q&A.

Run Test-Cluster with Disk Exclusions:

Test-Cluster -Node <node1>,<node2>,<node3> -Exclude "Storage"

If the issue only occurs when the disks are involved, you could try running Test-Cluster with the disk validation step excluded. This will bypass the storage validation part of Test-Cluster and prevent the disks from being tested, which may help avoid the disconnections during the test.

Ensure that the nodes within the cluster are capable of communicating with one another. Utilize the following PowerShell cmdlet to verify connectivity

Test-NetConnection -ComputerName <NodeName> -Port 445

Please follow the document for more information:

https://learn.microsoft.com/en-us/powershell/module/failoverclusters/test-cluster?view=windowsserver2025-ps
Jack 0 Reputation points

2025-03-14T08:29:08.11+00:00

Hi Suresh,

Thank you for the follow-up.

My concern is that I've never seen this issue occur before when running Test-Cluster without the storage exclude so while it seems likely it was the storage check that triggered the issue, we can't be sure.

It makes running a cluster validation against a cluster in Azure, say when adding a new node to a cluster, very risky in a live environment.

I'd feel more confident if this could be confirmed as a known issue / expected behaviour. We can then make sure that any work on a cluster is done in a maintenance window. Though this is still less then ideal.

Kind regards,

Jack
Marcin Policht 50,570 Reputation points MVP Volunteer Moderator

2025-03-14T17:45:15.08+00:00

Your expectations are reasonable - unfortunately, AFAIK, this is not that clear-cut. Refer to https://straightpathsql.com/archives/2020/06/cluster-validation-beware/ for more information

If the above response helps answer your question, remember to "Accept Answer" so that others in the community facing similar issues can easily find the solution. Your contribution is highly appreciated.

hth

Marcin
Jack 0 Reputation points

2025-03-17T13:45:21.5766667+00:00

Hi Marcin,

Thank you for the link, it certainly describes a similar scenario. This scenario does however describes ISCSI disks. While I believe Azure uses ISCSI in the background, the disks on the VM themselves do not present as ISCSI i.e. when running command Get-Disk | Select-Object Number, FriendlyName, BusType .

So I wouldn't expect the same scenario to apply. Is this potentially an explanation to why this doesn't occur every time the Test-Cluster command is run in my environment? The article seems pretty clear cut that it will always happen.

Kind regards,

Jack
Anonymous

2025-03-19T11:36:17.4566667+00:00

Hi Jack,

The Test-Cluster command conducts a series of validation tests that can exert pressure on the cluster's resources. If the disks are experiencing significant load or if there are other resource limitations during the test, this may result in unforeseen behavior, including temporary disconnections. Additionally, the condition of the cluster and the configuration of the nodes can influence the outcomes of the Test-Cluster command. Any inconsistencies or issues within the cluster configuration may lead to unexpected results during the validation process.

And, if you have any further query do let us know.

Thank you.
Marcin Policht 50,570 Reputation points MVP Volunteer Moderator

2025-03-19T11:39:50.41+00:00

Unfortunately, AFAIK, this behavior is not limited to iSCSI disks, as the response from Suresh seems to confirm.

If the above response helps answer your question, remember to "Accept Answer" so that others in the community facing similar issues can easily find the solution. Your contribution is highly appreciated.

hth

Marcin

Answer 2

Jack 0

Everything points to ensuring the Storage tests are not run as part of cluster validation when using Azure Managed Disks. While confirming this resolves the issue will be tricky due to me having run cluster validation with storage checks in Azure environments for several years without issue. I will carry out some further testing with and without storage tests to try and get some peace of mind.

Thanks for the responses on the question. I will update this thread if I find anything further from testing.

Arko 4,150 Reputation points Microsoft External Staff Moderator

2025-03-21T12:59:24.68+00:00
Hello Jack, The MVP Marcin Policht advice and documentation is spot on: don’t run full storage tests on live or production-managed disks unless necessary.

You may had initial success, but it doesn’t invalidate the risk. It just means that this issue can be situational or intermittent.

I would recommened you to try this approach to validate if storage is the root cause. Run Test-Cluster without storage tests i.e. Test-Cluster -Node node1,node2,node3 -Ignore "Storage" or if testing via ui then use “Run only tests I select”, and deselect all storage-related checks.

Secondly run Test-Cluster with only specific, known-safe disks

Test-Cluster -Include "List Storage Tests Here" -Disk "Cluster Disk 1"

Monitor for Event 157. If it doesn’t show up without storage checks then you’ve isolated the trigger. If it does then there might be another cause (e.g. transient disk latency in Azure, disk caching, VM host behavior).
Suresh Chikkam 2,135 Reputation points Microsoft External Staff Moderator

2025-03-24T05:31:59.92+00:00

Hello Jack,

Following up to see if the above comment was helpful. If this answers your query, do click thumbsup, if you have any further query do let us know.
Arko 4,150 Reputation points Microsoft External Staff Moderator

2025-03-26T06:04:11.1433333+00:00

Hello Jack, following up on this. Kindly confirm if you were able to isolate the test and narrow down on the issue. If your issue is resolved. Kindly update it here in answer for others in the community to benefit from it.

Share via

Azure Storage Disks Disconnected from Virtual Machine When Running Test-Cluster Command

2 answers

Your answer