iSCSI disks keep rearranging after reboot

Question

I'm hoping someone can help, I'm at my wits' end.

My firm has 2 Windows Server 2022 STD servers acting as Hyper-V hosts.

There is no clustering/failover.

They both connect to our SAN by iSCSI Initiator.

For every VM, there is a SAN volume connected to the host, the disk is online and mounted to a drive letter, the VHDX for the guest OS drive is on this volume.

For many VMs, there is at least one (often several) additional SAN volumes, connected to the host as "offline" then connected as a "physical" disk in the VM settings.

Every time the host is rebooted, the disks re-arrange themselves and the disks end up on the wrong VMs.

For example:

ServerA - OS Disk is a vhdx on D drive (Disk1 in the Host OS, Vol1 on the SAN), Data disk is offline disk connected as a Physical disk (Disk2 in the Host OS, Vol2 on the SAN).

ServerB - OS Disk is a vhdx on E drive (Disk3 in the host OS, Vol3 on the SAN), Data disk (1) is offline disk connected as a Physical disk (Disk4 in the host OS, Vol4 on the SAN), and Data disk (2) is offline disk connected as a physical disk (Disk5 in the host OS, Vol5 on the SAN).

Then we reboot the host.

Maybe the D drive is correct and has ServerA's OS disk. But now Disk2 in the Host OS may no longer be SAN Vol2... maybe it's SAN Vol3, which should have mounted as the E drive but is offline and attached inside ServerA.

E drive where ServerB's OS VHDX should be might be SAN Vol2, ServerA's data disk. Additionally it is trying to attach host OS Disk4 as a data disk, but that might be Vol5 now for some reason.

In this case ServerA starts up but doesn't have the right data and ServerB can't start.

This keeps happening, and we have way more than 2 VMs, and some have 5 or 6 passthrough volumes attached for data drives and I spend half a day untangling them to get the VMs running again.

All of the iSCSI connections are in the favorites and we used the "Auto Configure" button on "Volumes and Devices" tab in iSCSI initiator.

When I use Powershell to get-iscsisession all of the sessions show as persistent.

The disks are in different orders with each reboot. Sometimes some will stay the same, other times they'll be all over the place.

We used to have the exact same set up in 2019 and did not have this issue. IDK if there's a difference I'm not aware of between the two, or if I'm just doing something way wrong, or if something is truly broken.

Answer 1

Hello there,

If iSCSI disks keep rearranging after a reboot, it's likely due to the way the operating system handles storage devices and their connection order during the boot process. When the system restarts, it might assign different device names or paths to the iSCSI disks, resulting in the rearrangement.

Here are some potential reasons and solutions to address this issue:

Multiple iSCSI Targets: If you have multiple iSCSI targets, the system might not consistently identify them in the same order during the boot process. This can lead to different device names being assigned to each target.

Solution: To resolve this, you can use iSCSI initiator persistent binding. This feature allows you to create persistent configurations that associate specific iSCSI targets with specific iSCSI device names. Consult your operating system's documentation on how to set up persistent bindings for iSCSI.

Delayed Mounting: The system may try to mount iSCSI disks before the iSCSI initiators are fully initialized, leading to inconsistent device assignments.

Solution: Consider adding delays to the mounting process to allow enough time for the iSCSI initiators to establish connections before mounting the disks. This can be achieved through configuration files or scripts specific to your operating system.

Network Configuration: Issues with network configuration, such as slow network connectivity or improper network settings, can cause delays in iSCSI connection establishment.

Solution: Check your network settings, including network speed, network adapters, and IP configurations. Ensure that the network connectivity is stable and that the iSCSI targets are reachable before the system attempts to mount them.

udev Rules: The udev system in Linux can dynamically assign device names to storage devices based on various parameters, leading to changing device names on each boot.

I used AI provided by ChatGPT to formulate part of this response. I have verified that the information is accurate before sharing it with you.

Hope this resolves your Query !!

--If the reply is helpful, please Upvote and Accept it as an answer–

Answer 2

Hey,

If I were you, I would configure Failover Cluster, since you have all the hardware needed for it.

In any case, I would recommend you to connect iSCSI LUNs to the node it is used by. In addition, instead of connecting drives to the node and then passing physical drive to the VM, you can connect iSCSI LUNs (data) directly inside of the VM. Drive letter should not change after reboot, however, DeviceID will change. That might be the reason of the issue you are facing. iSCSI LUNs connect to the host in different order with every reboot.

You can also test similar configuration with alternative iSCS targets. StarWind VSAN as an example. https://www.starwindsoftware.com/starwind-virtual-san

Cheers,     
Alex Bykovskyi     
StarWind Software     
Note: Posts are provided “AS IS” without warranty of any kind, either expressed or implied, including but not limited to the implied warranties of merchantability and/or fitness for a particular purpose.