Cluster Network Validation - fail UDP port 3343

Question

When running the cluster network validation test on 2 x HPE DL380 Gen10 fully patched and firmware/driver updated Windows Server 2019 (LTSC 1809) with Hyper-V role nodes (pre-cluster creation) it gets the error below:
Network interfaces s-test-01.assemblyni.gov.uk - LOM1Port1_Mgmt and s-test-02.assemblyni.gov.uk - LOM1Port 1_Mgmt are on the same cluster network, yet address 10.63.35.30 is not reachable from 10.63.35.31 using UDP on port 3343.

The above problematic NICs are 1Gbps and used for management, RDP etc and are the only NICs with default gateways set and are connected via a Cisco 3750 switch with no ACL or port security configured.
Each server also has a single NIC with dual 25Gbps ports which are directly connected with DAC cables as we do not currently have the 25Gbps switches.
All other NICs are vNICs created on a switch embedded team on each server that uses the dual port 25Gbps NIC.
What has been tried:

1. Firewall has been disabled on all profiles on both servers. No other FWs between the servers
2. Real-time monitoring has been disabled on both servers for Windows Defender which is the only AV used
3. Servers full patched with HPP SPP 2020-09, all Windows OS Updates and restarted several times
4. When I change the mgmt. nic on one server to be in a different subnet the validation test works but why? Also when you go to create the cluster it will ask for a cluster VIP address which needs to be in the same subnet across all servers and it only offers the mgmt. NIC IP address subnets I assume because they are the only ones with default gateway set?
   I can find plenty of similar articles but none that answers this scenario and I would really appreciate any help or advice please.
   Thanks
   Stu
   kk

Answer 1

For anyone interested I found the solution but I cannot tell you why this works.
As opposed to using a single physical onboard network port, I decided to try teaming 2 of the onboard 1Gbps network adapters and then create a virtual NIC and use it for the management traffic across both server nodes and whatya know, it flamin worked!? But WHY?
So I don't know if this is a Failover cluster requirement or why I couldn't create the cluster when using a single physical network port for management traffic. Specifically the problem being failing to communication over UDP on port 3343.
I have not read any article saying watch out dont do crazy stuff like that because it is not supported and the pre-requisite for a Microsoft 2019 Hyper-V cluster is you must use resilient virtual NICs for your management traffic.
I dont know if this makes sense to anyone and I would appreciate if anyone is able to explain this, please feel free to enlighten me and/or others :-)

To finish I have to thank MIco who did enlighten me on the multi-subnet cluster articles.
I would also like to thank Romain Serre whose article made me think to try using vNICs for management.
https://www.tech-coffee.net/2-node-hyperconverged-cluster-with-windows-server-2016/#comment-3732
I also found this article useful:
https://social.technet.microsoft.com/Forums/windowsserver/en-US/c3e15170-2a83-48a8-b671-efc2a9afe4cf/s2d-cluster-validation-fails-firewall-and-udp-port-3343

Answer 2

Hi，
Can each host ping each IP address of the other host?
I’ve seen some similar threads which fixed the issue by changing the subnet mask.
And there seems no downside to having management network on different subnets.

Thanks for your time!
Best Regards,
Mico Mi

-----------------------------

If the Answer is helpful, please click "Accept Answer" and upvote it.
Note: Please follow the steps in our documentation to enable e-mail notifications if you want to receive the related email notification for this thread.

Answer 3

Hi thanks Mico,
All hosts can ping the other using any of the IP addresses as shown below. Not sure how that Local Area Connection is there as it doesn't show in network control panel but does with the Get-NetIPAddress cmdlet.
Please note I have since tried changing the IP addresses used by the management NICs but both servers are still in same subnet as you can see below but the problem still applies. If I put them in different subnets then cluster validation succeeds but why?
If the management network is on different subnets for each server then which IP address do you choose for the Cluster Network Object VIP considering that if it’s not in a common subnet to all server hosts I assume that only the server with an IP address in that subnet can own the VIP? Is that not right?

PSComputerName ipaddress interfacealias interfaceindex prefixlength

---

s-test-01 169.254.108.98 Local Area Connection* 10 4 16
s-test-01 192.168.50.10 LOM1Port1_Mgmt 5 24
s-test-01 172.16.5.2 RTest 10 24
s-test-01 172.16.5.3 SMB1 2 24
s-test-01 172.16.6.3 SMB2 13 24

s-test-02 169.254.43.83 Local Area Connection* 1 14 16
s-test-02 192.168.50.20 LOM1Port1_Mgmt 9 24
s-test-02 172.16.6.4 RTest 3 24
s-test-02 172.16.6.5 SMB1 5 24
s-test-02 172.16.5.5 SMB2 19 24

Answer 4

Hi,
Please check these docs:
Configuring IP Addresses and Dependencies for Multi-Subnet Clusters
Configuring IP Addresses and Dependencies for Multi-Subnet Clusters - Part II
Configuring IP Addresses and Dependencies for Multi-Subnet Clusters - Part III

You can configure cluster Virtual IP addresses in different subnets than the physical IP addresses of the Cluster Members.
The network "sees" the cluster as one Security Gateway that operates as a network router. The network is not aware of the internal cluster structure and physical IP addresses of Cluster Members.
Advantages of using different subnets:
You can create a cluster in an existing subnet that has a shortage of available IP addresses.
You use only one Virtual IP address for the cluster. All other IP addresses can be on other subnets.
You can "hide" physical Cluster Members' IP addresses behind the cluster Virtual IP address. This security practice is almost the same as NAT.

Best Regards,
Mico Mi

Answer 5

@Mico Thanks very much for your update and those links, very useful! I’m working my way through them and getting ready to implement it. I have a couple more questions to make sure I comprehend it:

1. Our current 2012 R2 Hyper-V cluster hosts (4 of them) are all on the same management subnet including the VIP. Is it no longer possible with 2019 to create a Hyper-V cluster with all nodes on the same management subnet along with the Cluster IP address? If it is possible then why are we getting the validation hard fail error with UDP 3343? Do the mgmt. NICs need to be teamed to ensure no single point of failure maybe? I’ve no idea why this isn’t working?
2. I am going to try the multi-subnet cluster you have mentioned. I have skimmed through the articles, is it correct to say that you assign each server management NIC an IP address in a different subnet and then when creating the VIP, tell the cluster 2 different IP addresses that it can use, one in each of the server management NIC subnets. So whichever server owns the Cluster Access Point (CAP), the VIP will be on that management subnet and if there is a failover the VIP will switch automatically to the pre-configured cluster IP address that is in the subnet of the new server management NIC without downtime or interruption to service? Or maybe a blip in service?
3. Just to be totally clear, I assume it’s obviously not possible for the CAP IP address to exist in a subnet that none of the cluster server hosts are in unless maybe there is some sort of hardware load balancer with a NAT?

Kind Regards
Stuart