Are vSwitches needed in a S2D Switchless Topology?

Jacob Petrie 1

I'm currently testing a 3-Node Switchless S2D Cluster. I was following the guide I found from Dell deployment-instructions-for-four-node-dual-link-full-mesh-switchless-interconnect which left me with each of the nodes connected and each link (dual links connecting each node) with a separate subnet and I can ping each node from the others (That's all good). Then it made one vSwitch that is connected to the external ports connecting to the rest of my network. The question being, is that how it's supposed to be set up, or do I need a vSwitch on each nic connecting the nodes together? This document S2D%20WS2016_ConvergedNIC_Configuration.docx gets referenced on setting up networking for S2D but it is written like you are just testing and I can't tell if the document is for actually deploying it or just understanding how it works theoretically (It also never brings up info about SMB traffic going over certain nics because it assumes a switch)

When created the cluster, the Test-Cluster each node came back with an error:

Unable to verify that all all adapters associated with SET Switch 'Management' on Node 'Hyperv3.domain.com' are of the same RDMA Technology type, some adapters are reporting 'Device Default' or not reporting data at all.

Which is that vSwitch is the built in Broadcom NetXtreme E-Series Advanced Dual-port 10Gb nics and not the QLogic FastLinQ QL41262-DE 25GbE nics that I'm using to connect the nodes. So that's not a real problem right?

Essentially, is there a way to tell it what nics to use or to verify that the cluster is actually using SMB with RDMA over the direct attached cables and not all just connecting to the 10Gb nic and through the switch without RDMA? Like it's currently working, and it's moving the VMs around when I down a node, I just want to make sure it works the way it should be.

System info: three Dell AX-640 servers
OS: Windows Server 2022 Datacenter
Network: two QLogic FastLinQ QL41262-DE 25GbE, Broadcom NetXtreme E-Series Advanced Dual-port 10Gb SFP+ and two Broadcom NetXtreme Gigabit Ethernet ports
CPU: two Intel Xeon Silver 4215R
Starage: a 240GB BOSS card, two 900GB SAS SSDs, and four 2.4TB SAS HDD
RAM: 192GB

using iWARP if that's important
this is basically a sanity check, didn't realize how little documentation there was on switchless setups when we purchased these

Ben Thomas 6 Reputation points MVP

2022-01-30T21:38:58.537+00:00

So just to confirm, you've cabled it like this?

If you run Get-NetAdapterRDMA on each node, do they all show the same result? The error message you're getting suggests that on 'Hyperv3.domain.com' there is a Broadcom NIC with RDMA enable or not identifiable compared to the rest.

Have you installed all the required drivers for the network adapters on the nodes?
Jacob Petrie 1 Reputation point

2022-01-30T22:12:37.98+00:00

Yeah, that's how it's set up. The Blue being the DAC cables (Plugged into QLogic NICs) and the pink being the cables to the switch (the Broadcom NICs).

Each of the QLogic ones come back with it enabled and are good, (I only posted the one but each node has that problem). This is the result that they all have.

So it's not showing the Broadcom NICs at all. But since I'm only using the Broadcom ones connecting to the rest of the network and not intended for the S2D storage backbone, does that matter if those NICs can't do RDMA? Or am I getting that error because it thinks that's the only interface it can use since it's the only vSwitch on the nodes?
Ben Thomas 6 Reputation points MVP

2022-02-02T19:48:17.677+00:00

Correct, you don't need RDMA on the broadcom NICs. RDMA improves S2D Storage Traffic/Latency/Perf and Live Migrations, not general traffic.

As long as your drivers are up to date, you should be fine.

Are you using RoCE or iWARP out of interest?
Jacob Petrie 1 Reputation point

2022-02-02T19:52:24.87+00:00

Awesome, thanks.

I'm using iWARP
Thomas Pike 1 Reputation point

2022-02-09T23:41:25.207+00:00
I am running a very similar setup but with two Dell R740XD ReadyNodes. The two 25Gb adapters are directly connected to the other node via DACs but I have some confidence that the 25Gb interfaces are not being used.

Functionally the nodes are working correctly apart from the poor I/O. All virtual machines are working correctly and migrate successfully between nodes.

Full network connectivity between nodes (management interfaces and storage interfaces can ping each other)

Test-RDMA.PS1 run on the storage interfaces passes using iWARP for both interfaces.

All servers running latest Windows and Dell firmware/driver updates.

"Get-SmbClientNetworkInterface" shows both storage interfaces with both "RSS Capable" and "RDMA Capable" as true. All other interfaces show "false" for both.

Counters and SNMP show very low usage of the storage interfaces (100kB/s) even when the nodes are under heavy I/O, but high network throughput through the 10Gb host interfaces.

Would appreciate any hints as to how to troubleshoot this further to get the nodes to use the storage interfaces for storage traffic.
Jacob Petrie 1 Reputation point

2022-02-18T19:48:05.22+00:00

Was doing some more research on this, and I found this blog post that might help failover clustering networking basics and fundamentals

Basically if you run Get-ClusterNetwork you should see the metric that the cluster uses to determine how it handles traffic, the lower the metric the higher priority for cluster traffic. To lower them you can specify that interface to be used for "Cluster only", which will use it over one set to "Cluster and Client"

Afterwards I was able to use Windows Admin Center and some basic performance counters on the NICs to observe that the traffic spiked on a live migration on the storage NICs

Another thing I picked up here Optimizing Hyper-V Live Migrations on an Hyperconverged you can tell live migrations to go over SMB using Set-VMHost -VirtualMachineMigrationPerformanceOption SMB (Note: when I did that I was already seeing traffic going over the correct NICs but the amount of traffic almost halfed, not sure why that's the case but whatever)

I'll be looking into this further as I still have a few issues with other parts of my cluster, I'll be sure to update you if I find new info. Hopefully that helps you some, and maybe we'll both get a cluster that is optimized :D

Share via

Are vSwitches needed in a S2D Switchless Topology?

Your answer