2016 on-link and default gateway behaviour

Jon Marshall 81 Reputation points
2021-09-29T14:53:15.217+00:00

I am seeing some very weird behaviour in Server 2016 routing.

We have 3 servers and each has 3 NICs -

1) Management Network - 192.168.5.128/27 (vlan 36)

2) Storage Network 1 - 192.168.5.192/27 (vlan 37)

3) Storage Network 2 - 192.168.5.224/27 (vlan 38)

The management network is on a vswitch and has a default gateway of 192.168.5.129 and each storage network does not have a default gateway set.

Each server can ping the other servers on the management IPs and the Storage Network 1 IPs but they cannot ping each other on Storage Network 2. Cabling and vlan assignment on the switches
IP settings and subnet masks have been checked, . The mac addresses of the NICs have been cross checked with the switch ports to ensure correct vlan etc.

The routing table on each server shows each interface as on-link so it knows they local interfaces.

If I traceroute from server 1 to server 2 on the storage Network 3 IPs then it -

1) first time it simply times out and shows destination host unreachable

2) second time the traceroute goes via the default gateway instead.

Does anyone know why it tries to use the default gateway because that does not make sense to me. The NIC is up, the port is in the correct vlan on the switch so why does the OS decide
to use the default gateway, this is not how routing normally works.

Could this behaviour be the cause of the loss of connectivity in the first place ie. the servers used to be able to communicate on Storage Network 2 (and have been for over a year) but they did lose connectivity and
now cannot communicate.

It is almost as if the servers have decided they can no longer connect on the locally attached interface (even though it is up) they will try the default gateway and stick with that.

Any insight much appreciated

Windows Server 2016
Windows Server 2016
A Microsoft server operating system that supports enterprise-level management updated to data storage.
2,382 questions
0 comments No comments
{count} votes