Windows Failover Custer Virtual Machines, using VMware and RDM for share disk

Question

Windows Failover Custer Virtual Machines, using VMware and RDM for share disk

Lanky Doodle 241

Hi,

We have both VMware and Hyper-V. We use VMware for some apps that aren't validated on Hyper-V. One of these uses Failover Clustering.

We have 2 Server 2019's using a shared disk, which is presented as an RDM from VMware. This app doesn't support CSV so only one node can see the disk at a time.

This app also uses a Generic Script. What we are seeing is during failover, the script fails to run automatically - it errors out. However if we manually run it after a failover, it works ok. So it seems to be that the script is trying to failover quicker than disk.

So we re-checked all the dependencies and made sure the script was dependent on the disk. No difference.

Running out of ideas!

Thanks

Shashank Singh 6,251 Reputation points

2021-01-14T08:32:50.143+00:00

So the issue is WSFC loosing connection to node or is it the generic script not working ?
Lanky Doodle 241 Reputation points

2021-01-14T10:59:01.593+00:00

It seems to be that during failover (manual or auto):

Everything seems to failover OK
The Generic Script fails because of the Disk (it's set to Auto Start)
A few seconds later we can manually start the Script

The fact we can manually start the script a few seconds after failover kind of proves that the disk is failing over. So it could potentially be a race condition. The dependency report looks good for the Generic Script and shows the disk as a dependency.

Can we set the Generic Script to be Delayed Start?

Thanks again
Lanky Doodle 241 Reputation points

2021-01-19T12:42:20.747+00:00

I've done some more digging:

The Generic Script itself is just a .VBS which starts a service, and that service itself brings the applications services up in the right order.

It seems that when the disk fails over during a restart of the current owner node, it's going over OK, going offline, then coming back online, which looks to be causing the VBS to halt.
Lanky Doodle 241 Reputation points

2021-01-19T12:42:31.36+00:00

If I simulate failover with either pausing a node or stopping the cluster service then everything is 100% fine. If I set the Generic Script to No Auto Start, it fails over properly (even during a restart) with the disk to the surviving node but stays stopped. If I then manually start the Generic Script it brings the disk online and everything else happens successfully.

With Auto Start on (either low, medium, high) the first process the VBS calls is ghosted and the applications services don't ever start. The fact the VBS launches at all suggests the disk at least appears online to the surviving node but is then lost again. And then comes back.

What checks can I do / where can I look to see if the disk is moving, going offline, then coming online?

2 answers

Your answer

Shashank Singh 6,251 Reputation points

2021-01-14T08:32:50.143+00:00

So the issue is WSFC loosing connection to node or is it the generic script not working ?
Lanky Doodle 241 Reputation points

2021-01-14T10:59:01.593+00:00

It seems to be that during failover (manual or auto):

Everything seems to failover OK
The Generic Script fails because of the Disk (it's set to Auto Start)
A few seconds later we can manually start the Script

The fact we can manually start the script a few seconds after failover kind of proves that the disk is failing over. So it could potentially be a race condition. The dependency report looks good for the Generic Script and shows the disk as a dependency.

Can we set the Generic Script to be Delayed Start?

Thanks again
Lanky Doodle 241 Reputation points

2021-01-19T12:42:20.747+00:00

I've done some more digging:

The Generic Script itself is just a .VBS which starts a service, and that service itself brings the applications services up in the right order.

It seems that when the disk fails over during a restart of the current owner node, it's going over OK, going offline, then coming back online, which looks to be causing the VBS to halt.
Lanky Doodle 241 Reputation points

2021-01-19T12:42:31.36+00:00

If I simulate failover with either pausing a node or stopping the cluster service then everything is 100% fine. If I set the Generic Script to No Auto Start, it fails over properly (even during a restart) with the disk to the surviving node but stays stopped. If I then manually start the Generic Script it brings the disk online and everything else happens successfully.

With Auto Start on (either low, medium, high) the first process the VBS calls is ghosted and the applications services don't ever start. The fact the VBS launches at all suggests the disk at least appears online to the surviving node but is then lost again. And then comes back.

What checks can I do / where can I look to see if the disk is moving, going offline, then coming online?

Answer 1

Hi,

Since your environment is mixed, we have limited experience in this scenario, it's recommended to open a case with MS, so that they can access your environment for troubleshooting.

Below is the link to open a case with MS:

https://support.microsoft.com/en-us/gp/customer-service-phone-numbers

Thanks for your time!
Best Regards,
Anne

-----------------------------

If the Answer is helpful, please click "Accept Answer" and upvote it.

Note: Please follow the steps in our documentation to enable e-mail notifications if you want to receive the related email notification for this thread.

Answer 2

Xiaowei He 9,936

Hi,

According to your recent feedback, it seems the Generic Script failover due to dependence disk failover, and after failover, the generic script won't start automatically.

Also from your original post, it seems the cluster shared storage is complex, that "presented as an RDM from VMware. This app doesn't support CSV so only one node can see the disk at a time."

Would you provide some screenshot of the Cluster disk used for the Generic Script in the Cluster disk and show use the sceenshot of the Gerneric Script resource. Besides, please also check the cluster log about the disk issues.

Thanks for your time!
Best Regards,
Anne

-----------------------------

If the Answer is helpful, please click "Accept Answer" and upvote it.

Note: Please follow the steps in our documentation to enable e-mail notifications if you want to receive the related email notification for this thread.

Lanky Doodle 241 Reputation points

2021-01-21T10:06:32.857+00:00

The Generic Script does start automatically during failover and the Disk comes online automatically during failover.

However the Generic Script just launches a VBS script that starts the main app service and then this service starts sub-services. The main app service starts ok as shown in Process Explorer but the sub-services end up as ghost processes and the app doesn't start.

If I set the Generic Script to No Auto Start, do a failover then manually start the Generic Script, everything works perfectly. It only seems that if the Generic Script is set to Auto Start that this problem exists; with it set to Auto Start, it causes an error in the event logs that the "Disk has been unexpectedly lost". But after this when I check Cluster Manager the disk has failed over to the other node AND is online.

So I can only assume that something is causing the disk to go online > offline > online in quick succession when the Generic Script is set to Auto Start

Share via

Windows Failover Custer Virtual Machines, using VMware and RDM for share disk

2 answers

Your answer