Hyper-V 2016, 5120 CSV has entered a paused state due to status_io_timeout

Question

Hyper-V 2016, 5120 CSV has entered a paused state due to status_io_timeout

Ryan 1

Looking for help as our issue reoccurs every 1-2 days over the past 2 months. We have already enabled jumbo frames on iscsi ports, switch and san. Also enabled VLT on our 2 Dell s4048t switches. Firmware and patches are up to date. No luck resolving thus far.

Issue: Every few days our hyper-v hosts are acting up and becoming somewhat unstable all while our guest VMs sporadically reboot and become unstable.

Environment:-
Cluster – Non S2D
Node – 4 Nodes
VM- Hyper V
Storage – ISCSI SAN (Dell Compellent)
OS – 2016
Hardware: Dell r730s
AV - Defender
Backups - MABS/DPM (guest vm level only)

Event IDs 5120, 5142, 1069, 1146 and 1230. Mainly 5120 and the CSVs are entering a paused state due to IO_STATUS_TIMEOUT

Also see this in cluster log: Microsoft Failover Cluster Virtual Adapter (NetFT) has missed more than 40 percent of consecutive heartbeats

No solution, the only thing to remediate the issue is powering down guest vms to remove load on the hosts (note that our hosts are underwhelmed from a memory and cpu standpoint)

Ryan 1 Reputation point

2022-03-17T03:39:34.237+00:00

Stress testing has been done on the network, cpu, ram etc and issue cannot be reproduced. Draining a node will reproduce the issue and we see tons of retransmits on the management networks (non-iscsi).

Dell has cleared the switches and san and feels it may be a hyper-v nic issue but we have tried many things already with no luck via Dell.

We run intel x550 10G NICs if I recall.

No AV is installed on these hosts

No correlation with DPM backups as we have those running constantly. Definitely can rule that out.

Cluster validation wizard all successful

No firewall or switch QOS occurring
Kevin Roggeman 0 Reputation points

2025-01-04T07:36:01.2433333+00:00

I have the same problem

We have two hyper v servers in failover cluster , and 3 weeks an arrow the clusterstorage goes to paused state

Always in the weekend between 5-7 o'oclock

We enable jumbo frames on issci ports on switch , disable backup already

But I think it's my file server because this machine was 3 times on HV03 node
Can I check somethings on the file server why I have many i/o througt ?

1 answer

Your answer

Ryan 1 Reputation point

2022-03-17T03:39:34.237+00:00

Stress testing has been done on the network, cpu, ram etc and issue cannot be reproduced. Draining a node will reproduce the issue and we see tons of retransmits on the management networks (non-iscsi).

Dell has cleared the switches and san and feels it may be a hyper-v nic issue but we have tried many things already with no luck via Dell.

We run intel x550 10G NICs if I recall.

No AV is installed on these hosts

No correlation with DPM backups as we have those running constantly. Definitely can rule that out.

Cluster validation wizard all successful

No firewall or switch QOS occurring
Kevin Roggeman 0 Reputation points

2025-01-04T07:36:01.2433333+00:00

I have the same problem

We have two hyper v servers in failover cluster , and 3 weeks an arrow the clusterstorage goes to paused state

Always in the weekend between 5-7 o'oclock

We enable jumbo frames on issci ports on switch , disable backup already

But I think it's my file server because this machine was 3 times on HV03 node
Can I check somethings on the file server why I have many i/o througt ?

Answer 1

Limitless Technology 40,076

Hello @Ryan

There could be some Latency between your CSV Storage network and Hyper-V network.
Please check if you have any File Server , SQL server or any application server which requires frequent use of Storage access which can lead to I/O bottleneck
Please Disable any Antivirus program you may have for temporary purpose.
Please check if you have any QoS at firewall or Switch or Dell storage level level which is not allowing full traffic flow between storage , host and VMs.
Please try to disable Time of DPM back up during non-working hours or during weekend.
Please run Hyper-V Cluster validation wizard to check all cluster configurations are identical and there should be no warning or errors in the Cluster report.

Please have a look on below Microsoft article to troubleshoot the I/O issue in the Hyper-V Cluster.

https://techcommunity.microsoft.com/t5/failover-clustering/troubleshooting-cluster-shared-volume-auto-pauses-8211-event/ba-p/371994

-----
--If the reply is helpful, please Upvote and Accept as answer--

Ryan 1 Reputation point

2022-11-08T19:40:45.45+00:00
Hello @Limitless Technology

Dell and a 3rd party vendor have ruled out the SAN and Switches basically so cannot point to latency.

No IO bottleneck have been found currently

AV is disabled on the hosts already

No QOS enabled anywhere

DPM backups being disabled made no difference

Zero concerns in Hyper-V cluster validation wizard, all is healthy

Microsoft and Dell have both mentioned other customers are having this same issue but no answer has been uncovered. The only way things have gotten better is by retiring/migrating VMs to Azure.

Seems to be load sensitive but not load induced. Stress testing the Network, Hosts, SAN, etc did not reproduce the issue.
Ryan 1 Reputation point

2022-11-08T19:41:37.673+00:00
Hello @Limitless Technology

Dell and a 3rd party vendor have ruled out the SAN and Switches basically so cannot point to latency.

No IO bottleneck have been found currently

AV is disabled on the hosts already

No QOS enabled anywhere

DPM backups being disabled made no difference

Zero concerns in Hyper-V cluster validation wizard, all is healthy

Microsoft and Dell have both mentioned other customers are having this same issue but no answer has been uncovered. The only way things have gotten better is by retiring/migrating VMs to Azure.

Seems to be load sensitive but not load induced. Stress testing the Network, Hosts, SAN, etc did not reproduce the issue.

Share via

Hyper-V 2016, 5120 CSV has entered a paused state due to status_io_timeout

1 answer

Your answer