Node down but Cluster Manager still says "Up". Heartbeat/Health Monitoring gap?

Question

Guys, need some advice on a WSFC heartbeat anomaly.

1. Issue

We cut power to one node in our Hyper-V / SQL Cluster to test failover. The physical server was 100% dead, but Failover Cluster Manager still stubborn—showing the node status as "Up". Because of this stale state, the cluster didn't even attempt to reassign the cluster roles immediately. Quite jiam if this happens in production during an actual power outage.

2. Questions

Heartbeat Tweak: Are you guys running default cluster heartbeat settings on local subnets, or do you tighten SameSubnetThreshold / CrossSubnetThreshold?

Clustered Service State: Is there any chance clussvc hung on the remaining nodes, making them miss the ICMP/UDP 3343 heartbeat loss from the dead node?

Diagnostics: What's the best way to trace this in Cluster.log? Looking for HeartbeatPxe or NodeLost events, but catch no ball why the failover detection took forever to trigger.

Anyone encountered this kind of ghost node state? Let me know how you guys fixed it.

Answer 1

Hi Nurul Amin,

Most enterprise clusters rely on the native default settings, which enforce a SameSubnetThreshold of ten heartbeats with a one-second delay, resulting in a ten-second failure window. You can query your current configuration by opening an administrative PowerShell session and executing the command Get-Cluster | Format-List *Threshold*, *Delay*. While you can adjust these using the Set-ClusterParameter cmdlet to tighten the threshold to five heartbeats, lowering it excessively on busy virtualization hosts risks triggering false failovers from minor network congestion.

A cluster service hang across all surviving nodes is extremely rare during a power cut; the delay is almost always a function of UDP 3343 heartbeat packet timeouts. To investigate this accurately, generate a targeted diagnostic trace by executing Get-ClusterLog -TimeSpan 15 -Destination . immediately after an incident. Open the resulting text file and search for the tokens IsNodeReachable, NodeDown, or dead. Tracking these entries will expose the precise timing of the network timeout and clarify whether hardware link persistence caused the delayed failover initiation.

Hope this answer has brought you some useful information. If it did, please hit “accept answer”. Should you have any questions, feel free to leave a comment.

VPHAN