Virtual machines experiencing severe i/o slowdown on sofs

Question

Our virtual machines hosted on an sofs cluster have recently started experiencing severe i/o performance issues. We found that the underlying csv volumes are entering redirected access, and the problem appears to be related to incorrectly configured mpio paths. The storage connections are not being recognized consistently, which is causing the cluster to redirect disk traffic and significantly reduce performance.

We have already checked the cluster and storage configuration, but we are still unsure how to safely restore the correct mpio path bindings without disrupting the running virtual machines. Could someone please explain the recommended way to re-bind the mpio storage claims and bring the csv volumes back to normal direct access? Any guidance on checking the existing mpio claims and restoring the correct paths would be greatly appreciated.

Answer 1

Hello,

The redirected access you’re seeing on the CSV volumes is a direct consequence of the cluster nodes losing their optimal MPIO path bindings. When the cluster detects a path failure or inconsistent binding, it forces I/O through the coordinator node, which explains the severe performance degradation. The safest way to restore direct I/O without disrupting running VMs is to correct the MPIO configuration at the host level while leaving the CSVs online.

Start by confirming the current path state with mpclaim -v and Get-MSDSMGlobalDefaultLoadBalancePolicy. This will show you which devices are claimed by MPIO and how the paths are being balanced. If you see that the storage LUNs are not claimed under MSDSM, you need to rebind them. The supported method is to run mpclaim -r -i -d <HardwareID> for the storage devices in question, followed by a reboot of each node one at a time. Do not reboot all nodes simultaneously; drain roles from a node, reboot it, verify that CSV returns to direct I/O, then move on to the next.

Also check that the DSM load balance policy is set to “Round Robin With Subset” or “Least Queue Depth,” depending on your storage vendor’s recommendation. You can adjust this with Set-MSDSMGlobalDefaultLoadBalancePolicy -Policy RRWS or the appropriate value. Ensure that the registry key HKLM\SYSTEM\CurrentControlSet\Control\MPDEV reflects the correct bindings after the claim.

If you suspect the claims are inconsistent across nodes, run Get-MSDSMLocalClaimedHwId on each cluster node and verify that the same hardware IDs are claimed. Any mismatch will cause CSV to redirect. Once the claims are consistent and the load balance policy is correct, the cluster should automatically return the CSVs to direct I/O without manual intervention.

If after correcting the claims you still see redirected access, check the cluster event logs under Applications and Services Logs\Microsoft\Windows\FailoverClustering\Operational for Event IDs 5120 and 5142, which will tell you which node is failing path validation. That will help isolate whether the issue is still at the MPIO layer or if it’s a fabric zoning/target presentation problem.

I hope you've found something useful here. If it helps you get more insight into the issue, it's appreciated to accept the answer. Should you have more questions, feel free to leave a message. Have a nice day!

HL.

Answer 2

For CSV volumes that are entering redirected access because MPIO paths are inconsistent, the supported approach is to verify the current MPIO state, correct the path configuration, and then update or reinstall the MPIO claim configuration if needed.

Recommended sequence:

1. Check the current MPIO configuration and claims. Use these tools to review what Windows currently recognizes:
   • PowerShell: Get-MPIOAvailableHW and Get-MPIOSetting
   • Command line: mpclaim -e and mpclaim -r -i -a
2. Verify that all storage paths are healthy.
   • Confirm all MPIO paths are online.
   • In clustered environments, confirm all nodes can see the shared storage in Disk Management and Failover Cluster Manager.
   • If only one path is detected, rebuild the iSCSI connections.
   • For redundancy, each iSCSI connection should use a different network adapter.
3. Update the storage stack before rebinding claims.
   • Update storage, network, and multipath drivers to the latest supported versions from the hardware or storage vendor.
   • Update storage or SAN controller firmware as recommended. This is important because recurring Event ID 153 in virtual environments can reflect a Storport driver issue, and outdated drivers or firmware are a common cause of path instability.
4. Check for conflicting multipath software. If duplicate disks or changed disk numbering is occurring, make sure only one multipath solution is attached. Remove unsupported DSMs, restart, and then reinstall or reconfigure MPIO. If repeating events refer to third-party DSMs, consult the storage vendor or migrate to a supported native DSM.
5. Rebind the MPIO claims. The documented command for refreshing and applying claims is:

      mpclaim -r -i -a
   

   A restart may be prompted.
6. After the paths are restored, validate cluster behavior.
   • Confirm the disks return to normal visibility and are no longer unexpectedly offline, read-only, or deallocated.
   • Confirm CSV volumes leave redirected access and return to normal direct access.
   • If disk resources are slow to recover on the cluster, increase the cluster disk resource timeout:

      Get-ClusterResource "<Resource_Name>" | Set-ClusterParameter PendingTimeout 300000
   

   PendingTimeout is in milliseconds.
7. If redirected access is accompanied by CSV paused events such as 5120/5142, also check for non-MPIO causes. Common causes include:
   • network bottlenecks or teaming mismatch
   • physical disk or HBA failure
   • switch or cable faults
   • antivirus or unsupported filter drivers

Important caution:

• Rebinding claims with mpclaim -r -i -a can require a restart, so this should be planned as a maintenance action if virtual machines are running.
• Removing DSMs or reinstalling MPIO can temporarily affect disk visibility. Use caution in production clusters.

If the issue persists after claims are refreshed, the next checks are the physical path health, vendor DSM compatibility, and driver/firmware consistency across all cluster nodes.

---

References:

• Multipath I/O (MPIO) troubleshooting guidance
• Troubleshoot storage issues in Hyper-V and Windows Server failover clusters
• Event ID 5120 Cluster Shared Volume troubleshooting guidance