SMB Traffic moving to wrong NIC

Scott Guenther 6 Reputation points
2023-08-09T02:28:46.56+00:00

I have 6 servers, 2 DC, and 4 Nodes for a cluster. All have dual 1gb NICs teamed and Dual 40Gb(infiniband) Nics teamed. Ininiband network is completely separate, it's only for moving data between servers.

Teams are just for redundancy, dual switches for each team.

our old cluster had the same setup and we had no issues. however now all trafic goes over the 1gb nics no matter what we do.

1gb 172.16.50.x has a default gateway

40gb 192.168.1.x no default gateway or DNS, everything is done by IP to force over the appropriate NIC

I connect to a share over 192.168.1.x network yet all traffic defaults to the 1gb network(172.16.50.x netwrok) and maxes out at 119mbps (weirdly above 112 of a single connection but way below a teamed connection) The odd thing is neither NIC knows of the other subnet. it seems like SMB multichannel is messing with me.

Windows Network
Windows Network
Windows: A family of Microsoft operating systems that run across personal computers, tablets, laptops, phones, internet of things devices, self-contained mixed reality headsets, large collaboration screens, and other devices.Network: A group of devices that communicate either wirelessly or via a physical connection.
814 questions
0 comments No comments
{count} votes

5 answers

Sort by: Most helpful
  1. Scott Guenther 6 Reputation points
    2023-08-12T17:21:46.4466667+00:00

    Still no luck. any traffic to the 192.168 network is still going over the gigabit connections. no traffic will flow over infiniband. If I disable the Infiniband Nic I can no longer access the 192.168.x.x network yet it is never actually used when transferring data.

    the gigabit team has no clue the infinaband network even exists, they have no connection to each other and the infiniband network doesn't even have access to a router. its just plain switching for storage purposes. Yes the SM Manger is up ( we don't even get link at the servers when its not running)

    this is truly baffling as I don't even know how it finds the correct server when I input a 192.168.x.x address. the gigabit network has to be advertising it somehow.

    Since I have tried everything I can think of and all suggestions in these posts I am going to kill off 2 nodes and install server 2019 as that's what the old cluster used. Maybe this is something new in 2022.

    1 person found this answer helpful.
    0 comments No comments

  2. Limitless Technology 44,506 Reputation points
    2023-08-09T08:16:39.9733333+00:00

    Hello there,

    If your SMB (Server Message Block) traffic is moving to the wrong Network Interface Card (NIC), it could be due to several reasons. Here are some steps you can take to troubleshoot and resolve the issue:

    Network Configuration: Check your network configuration settings to ensure that the correct NIC is configured for SMB traffic. This includes checking IP addresses, subnet masks, gateway settings, and DNS configurations.

    NIC Priority: Most operating systems allow you to set the priority of network interfaces. Make sure that the NIC you want to use for SMB has a higher priority than the other NICs.

    Binding Order: Check the binding order of your network interfaces. The binding order determines the order in which network interfaces are used for communication. Make sure the desired NIC is at the top of the list.

    Route Table: Check the routing table of your system to ensure that the correct NIC is being used for the destination IP addresses relevant to SMB traffic.

    Firewall and Network Profiles: Windows, for example, has different network profiles (public, private, domain). Make sure the correct profile is assigned to the NIC that you want to use for SMB. Firewalls can also affect traffic flow, so adjust firewall rules accordingly.

    Network Card Settings: Check the settings of the NIC itself, including speed, duplex settings, and any advanced settings that might affect how traffic is routed.

    Network Metrics: Some systems use network metrics to determine the best route for traffic. Make sure the desired NIC has a lower metric value for SMB traffic.

    Network Teaming/Bonding: If you're using network teaming or bonding, ensure that the settings are correctly configured, and the load-balancing or failover settings are appropriate for your SMB traffic.

    Driver Updates: Ensure that you have the latest drivers installed for all your NICs. Outdated drivers might cause unexpected behavior.

    I used AI provided by ChatGPT to formulate part of this response. I have verified that the information is accurate before sharing it with you.

    Hope this resolves your Query !!

    --If the reply is helpful, please Upvote and Accept it as an answer--

    0 comments No comments

  3. Gary Reynolds 9,591 Reputation points
    2023-08-09T10:08:45.16+00:00

    Hi @scott Guenther

    If the machine you used to connect to the share via 192.168.1.x IP is not in the same network subnet (as defined by the server's subnet mask), then the traffic will be sent via the gateway of the 172.16.50.x.

    You can use route print to display the routing table to help identify which internet the traffic will use.

    Gary.

    0 comments No comments

  4. Scott Guenther 6 Reputation points
    2023-08-12T17:32:00.88+00:00

    OK. server 2019 performs as expected with no configuration change. I simply wiped 2 nodes and installed 2019 server and all 192.168.x.x trafic flows over the Infiniband network. seeing speeds of 2.3gbps (limited by a single SSD).

    It seems there is something new in server 2022 that is messing up the networking.

    when testing 2019 to 2022 server I still see all the traffic over the gigabit network (172.16.50.x) so it appears it only needs one side to be 2022 for the error to happen.

    I would still like to see if there is a fix for this as I would like to use 2022.

    0 comments No comments

  5. Ray Yano 0 Reputation points
    2024-01-11T20:55:16.5633333+00:00

    I face that problem in my environment too.

    Server:

    • Debian GNU/Linux 12 (bookworm)
    • samba 2:4.17.12+dfsg-0+deb12u1
    • Intel I211 Gigabit Network, Intel I211 Gigabit Network as bond0 (192.168.0.39/24)
    • Intel X540-AT2 10-Gigabit as br0 (192.168.100.39/24)

    Client:

    • Windows 11 Pro 23H2 22631.3007
    • Intel I211 Gigabit Network (192.168.0.? DHCP)
    • AQC107 10G (192.168.100.128/24)

    I inspect the connection in Wireshark. I ran this command.

    net use \192.168.100.39\shared

    And I got a packet

    SMB2 Ioctl Request FSCTL_DFS_GET_REFERRALS, File: \192.168.100.39\shared

    but the destination is 192.168.0.39. I think this problem occurred because of any host or authentication, connection caching function, but I cannot identify that.


Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.