This browser is no longer supported.
Upgrade to Microsoft Edge to take advantage of the latest features, security updates, and technical support.
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(208, 32%, 30%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKK%3C/text%3E%3C/svg%3E)
Hello,
we have a 2-node Failover Cluster on Windows Server 2019 S2D with only the 'File Server' role installed. The Role hosts a single SMB file share.
When pausing one node using the 'pause -> drain roles' command clients lose connectivity to the SMB share for a very long amount of time (5 min)
In the Log I can see this event in preparation to switch the disk to the other node:
00002234.00003344::2020/08/31-07:40:36.695 INFO [RES] Physical Disk <Cluster Virtual Disk (FileServerVirtualDisk)>: Sending FSCTL_DISMOUNT_VOLUME ...
However, it never succeeds. The cluster share is offline at this time as cluster ip and cluster name are already down.
About 3.5 minutes later, a timeout is reported:
00002234.00002258::2020/08/31-07:44:05.234 WARN [RHS - Timeout] Resource 'Cluster Virtual Disk (FileServerVirtualDisk)' has not responded to the call OFFLINERESOURCE:4. The timeout to respond has been exceeded by 16 milliseconds, taking recovery actions.
Another 4 minutes later, another timeout.
00002234.00003104::2020/08/31-07:48:05.221 ERR [RES] Physical Disk <Cluster Virtual Disk (FileServerVirtualDisk)>: Terminate: Terminate thread timed out, attempting to stop pool defense and detach space for device number 7.
After that, finally failover takes place. Thats expected behaviour, however, a timeout should never happen in the first place.
When I tried the switchover while there was no load on the file share, everything works as expected, e.g.
00002450.00000af4::2020/08/26-08:03:30.144 INFO [RES] Physical Disk <Cluster Virtual Disk (FileServerVirtualDisk)>: Sending IOCTL_VOLUME_OFFLINE ...
00002450.00000af4::2020/08/26-08:03:30.145 INFO [RES] Physical Disk <Cluster Virtual Disk (FileServerVirtualDisk)>: Offlining disk ...
Only 1ms after the IOCTL_VOLUME_OFFLINE command the next action happens.
Obviously, something is preventing IOCTL_VOLUME_OFFLINE to finish under load.
I have watched Task Manager -> Memory during that operation and I could see the amount of "Cached" Memory is decreasing while waiting for the IOCTL_VOLUME_OFFLINE command.
How can I make sure my File Server Failover cluster role can actually fail over without losing connectivity for >5 minutes?
/Klaus
have you run the cluster validation wizard against your configuration to see that the cluster is seeing all components without errors and warnings?
Yes, validation wizard is green.
Everything works perfect when not under load. Its only a problem when the system is actually used.
Hi,
According to your description and based on my experience, it may have RHS deadlock when you drain the node:
"RHS will sit there waiting for the resource to respond to an IsAlive call, and eventually it will give up and need to take recovery action. By default RHS will wait for 5 minutes for the resource to respond to an entry point call to it. This is configurable with the resource DeadlockTimeout common property.
To modify the DeadlockTimeout property of an individual resource, you can use the following PowerShell cmdlet command:
(Get-ClusterResource “Resource Name”).DeadlockTimeout = 300000"
You may check the following article for detailed information about RHS deadlock:
If the reply could be of help, please help to accept it as an answer, thanks for your cooperation!
Thanks for your time!
Best Regards,
Anne
Yes, I know about that, Thats the reason why I wrote thats the expected behaviour.
What I want to know is how to avoid that deadlock.
I feel like I am the first person who actually uses a fileserver cluster. In theory all is good but when you actually start using it it all falls apart.
I probably might need to change the deadlocktimeout to a much shorter number if no real solution comes up. However, I really do not know how that will impact the overall stability of the system.
Hi,
---What I want to know is how to avoid that deadlock.
Generally, please ensure the cluster nodes have installed the latest windows update, windows update may resolve some known issues.
---I probably might need to change the deadlocktimeout to a much shorter number if no real solution comes up. However, I really do not know how that will impact the overall stability of the system.
If after installing the latest windows update, you still have the deadlock issue, as a workaround, you may use the powershell command in the link I provided to shorter the deadlock time, such as short to 3 minutes, but it's not recommended to make the deadlock timeout less than minutes.
Thanks for your time!
Best Regards,
Anne
All Updates were already installed.
Three minutes still is much too long.
The thing is - we need a switchover time of less than a minute, Actually I did expect seconds. Thats why we did install a cluster in the first place. Even a failover of the service in case of a host failure is faster than a 'normal' maintenance move of the role.
Hi,
Even a failover of the service in case of a host failure is faster than a 'normal' maintenance move of the role.
That is, host failure will be detected immediately, however RHS deadlock needs to wait for the timeout.
If you only have the file share role in the cluster, you may try to make the timeout less than one minute. The reason not recommended to set the timeout too short is due to all resource runs in the same RHS, if some resource only has slow respond speed but still running, the RHS restart due to respond timeout will cause the cluster roles not stable.
As far as I'm concerned, if you only have one cluster role, set RHS timeout short will have little effect.
Thanks for your time!
Best Regards,
Anne
At least it will fail over more quickly.
I have still concerns what that means in terms of my data: Is everything lost that is still in cache? Is the data safe? How can I tell whether a data loss can occur or even did occur?