Network Droping every 2 hours

Question

Network Droping every 2 hours

Zachariah DeGraaf 1

I have a Windows 10 Pro box that is dropping Network connectivity every 2 hours for roughly 5-10 seconds. This box is a VM running on a Proxmox, which is a Linux KVM based hypervisor. I started to notice the issue after implementing adding this host to a Hypervisor managed VLAN, but it the issue doesn't appear to be related to the environment as I am not having issues with any of my other machines with the same network settings. I do have WSL on my machine, but I rarely use it.
I looked through the Application, Security, Setup and system Event logs and I am consistently receiving Event's 8019, 8033 at the same time as the disconnects occur. These are both DNS Client events so I am not sure if these could somehow be causing the disconnects or if they are just another symptom. Looking at the errors, they are failures to register the machines IP address updates with the domain controller, they are failing because the server they are sending the updates to isn't the domain controller. I have a somewhat complicated DNS setup using conditional forwarding with Linux Bind9 servers that I won't go into unless people think it's necessary, but I can confidently say it does work for resolution of Domain DNS records.

Any advice on how to fix this issue is appreciated

Gary Nebbett 6,476 Reputation points

2020-12-11T09:48:09.653+00:00

Hello @Zachariah DeGraaf ,

What does "dropping network connectivity" mean in detail? Are existing TCP connections terminated or is there just no measurable traffic flowing during this period?

How do you determine/notice that such an event has happened and estimate its duration (e.g. how do you estimate the "5-10" seconds for the duration of the event)?

If the "every 2 hours" estimate is reasonably reliable, you could enable increased tracing of network activity when an interruption is expected. The tracing that I am thinking of (the Event Tracing for Windows (ETW) Microsoft-Windows-TCPIP provider) captures lots of data, so one would not want to leave it running for too long.

Gary
Zachariah DeGraaf 1 Reputation point

2020-12-14T16:43:42.69+00:00

All connections to the VM are dropped. I usually am RDP'd to the box so I notice right away when everything seems to freeze up. I am estimating 5-10 seconds based on 2 pings dropping before it's reachable again.

It has always been 2 hours plus or minus 10 seconds since the last disconnect that this occurs. The exact time that this occurs each day varies though. For instance today it dropped at about 8:50:50 and 10:50:50. Other days it's dropped at 11:20.

Thank you for informing me of Event tracing, this is exactly the type of extra troubleshooting step that I was looking for. I've never used it before but after learning the ropes I found something that could be of interest. It appears that near or at the beginning of the disconnect, all of the routes for the local subnet are being deleted, here is one of four event messages:
IP: Route 0xffffa38f5e3fd670 deleted on interface 20, Protocol = IPv4, DestinationPrefix = 172.16.85.0 /24, Nexthop = 0.0.0.0 .
Zachariah DeGraaf 1 Reputation point

2020-12-14T16:46:42.97+00:00

The four events were messages about deleting routes for the default gateway, host IP, network IP and broadcast IP. After 7 seconds all of the routes are recreated. Not sure if this is another symptom of the problem or the cause. I've adjusted Event Tracing to track router advertisements to see if any of those might be causing problems.
Gary Nebbett 6,476 Reputation points

2020-12-14T17:18:57.13+00:00

Hello @Zachariah DeGraaf ,

"dropped" for me somehow has a sense of "closed" or "reset", but I guess that your TCP connections "survive" the network interruption and fully recover once the necessary routes are in place again.

Route creation/deletion is usually caused by something (e.g. interface status change, such as a new VPN connection), and hopefully that would be visible in the trace too.

One possibility is that an "application" (e.g. route.exe) is manipulating the routes - the process ID in the route delete/create events might be meaningful and useful under these circumstances.

One can also get stack traces for each of the events - this might be useful if the reason for the route deletion/creation cannot be deduced from the existing trace data.

Gary
S R 1 Reputation point

2020-12-14T17:41:51.303+00:00

In a non domain setup I would test this, so not sure if it is applicable to domain setup:

If you do a a time sync try after about an hour after last drop does it cause virtual nic to drop any connections like what your are seeing?

I assume this test still works: in cmd.exe
net time \yourhost /querysntp
net time \yourhost /set /y

Are all your other VMs using same time server?

Item 2:
Is your VM set to go to standby mode afetr 2 hours?

Item3:
Have you tested using a static IP address.

Time glitches could interupt connections and cause DHCP lease issues - it is unlikely but worth doing a test.

Have you tested with just hypervisor and one VM running + RDP connect from separate physical machine?

Scott

2 answers

Your answer

Gary Nebbett 6,476 Reputation points

2020-12-11T09:48:09.653+00:00

Hello @Zachariah DeGraaf ,

What does "dropping network connectivity" mean in detail? Are existing TCP connections terminated or is there just no measurable traffic flowing during this period?

How do you determine/notice that such an event has happened and estimate its duration (e.g. how do you estimate the "5-10" seconds for the duration of the event)?

If the "every 2 hours" estimate is reasonably reliable, you could enable increased tracing of network activity when an interruption is expected. The tracing that I am thinking of (the Event Tracing for Windows (ETW) Microsoft-Windows-TCPIP provider) captures lots of data, so one would not want to leave it running for too long.

Gary
Zachariah DeGraaf 1 Reputation point

2020-12-14T16:43:42.69+00:00

All connections to the VM are dropped. I usually am RDP'd to the box so I notice right away when everything seems to freeze up. I am estimating 5-10 seconds based on 2 pings dropping before it's reachable again.

It has always been 2 hours plus or minus 10 seconds since the last disconnect that this occurs. The exact time that this occurs each day varies though. For instance today it dropped at about 8:50:50 and 10:50:50. Other days it's dropped at 11:20.

Thank you for informing me of Event tracing, this is exactly the type of extra troubleshooting step that I was looking for. I've never used it before but after learning the ropes I found something that could be of interest. It appears that near or at the beginning of the disconnect, all of the routes for the local subnet are being deleted, here is one of four event messages:
IP: Route 0xffffa38f5e3fd670 deleted on interface 20, Protocol = IPv4, DestinationPrefix = 172.16.85.0 /24, Nexthop = 0.0.0.0 .
Zachariah DeGraaf 1 Reputation point

2020-12-14T16:46:42.97+00:00

The four events were messages about deleting routes for the default gateway, host IP, network IP and broadcast IP. After 7 seconds all of the routes are recreated. Not sure if this is another symptom of the problem or the cause. I've adjusted Event Tracing to track router advertisements to see if any of those might be causing problems.
Gary Nebbett 6,476 Reputation points

2020-12-14T17:18:57.13+00:00

Hello @Zachariah DeGraaf ,

"dropped" for me somehow has a sense of "closed" or "reset", but I guess that your TCP connections "survive" the network interruption and fully recover once the necessary routes are in place again.

Route creation/deletion is usually caused by something (e.g. interface status change, such as a new VPN connection), and hopefully that would be visible in the trace too.

One possibility is that an "application" (e.g. route.exe) is manipulating the routes - the process ID in the route delete/create events might be meaningful and useful under these circumstances.

One can also get stack traces for each of the events - this might be useful if the reason for the route deletion/creation cannot be deduced from the existing trace data.

Gary
S R 1 Reputation point

2020-12-14T17:41:51.303+00:00

In a non domain setup I would test this, so not sure if it is applicable to domain setup:

If you do a a time sync try after about an hour after last drop does it cause virtual nic to drop any connections like what your are seeing?

I assume this test still works: in cmd.exe
net time \yourhost /querysntp
net time \yourhost /set /y

Are all your other VMs using same time server?

Item 2:
Is your VM set to go to standby mode afetr 2 hours?

Item3:
Have you tested using a static IP address.

Time glitches could interupt connections and cause DHCP lease issues - it is unlikely but worth doing a test.

Have you tested with just hypervisor and one VM running + RDP connect from separate physical machine?

Scott

Answer 1

Hello @Zachariah DeGraaf ,

By way of illustration, this is an example of a route addition with an obvious cause:

And this is an example of a route added by an application (the RasMan service, process ID 5392):

The stack at this point looks like:

tcpip!IppLogRouteChangeEvents+0x382
tcpip!IppValidateSetAllRouteParameters+0x81274
tcpip!IppUpdateUnicastRouteUnderLock+0x7c
tcpip!IppAddAutoConfiguredRoutesForAddress+0x10b
tcpip!IppDadComplete+0x60ca8
tcpip!IppProcessDadCompletion+0x15
tcpip!IppRestartDad+0x606cd
tcpip!IppFindOrCreateLocalAddress+0x3aa
tcpip!IppCommitSetAllLocalAddressParameters+0x5bdb7
tcpip!IppSetAllLocalAddressParameters+0x3f
tcpip!IpSetAllLocalUnicastAddressParameters+0x10
netio!NsiSetAllParametersEx+0x13a
nsiproxy!NsippSetAllParameters+0x1f4
nsiproxy!NsippDispatch+0x200
ntoskrnl!IofCallDriver+0x55
ntoskrnl!IopSynchronousServiceTail+0x1a8
ntoskrnl!IopXxxControlFile+0x5e5
ntoskrnl!NtDeviceIoControlFile+0x56
ntoskrnl!KiSystemServiceCopyEnd+0x25
ntdll!NtDeviceIoControlFile+0x14
NSI!NsiIoctl+0x15a
NSI!NsiSetAllParametersEx+0x35
rasppp!SetAllParameters+0x78
rasppp!SetIPAddressOnInterface+0x9e
rasppp!ApplyIpcpSettings+0x69a
rasppp!IpcpProjectionNotification+0x65e
rasppp!NotifyIPCPOfProjection+0xaf
rasppp!FsmThisLayerUp+0x2c5

Gary

Answer 2

Anonymous

Hi,

Thanks for posting in Q&A platform.

May I know if there is any other error message when network was dropped? Or did you made any change on the impacted win 10 client?

The problematic NIC is a wired NIC or a wireless NIC? Have you tried update the problematic network adapter driver to the latest version or choose a suitable historical version?

I would also suggest you to disable 3rd party anti-virus software temporarily to test if the issue still existed.

Best Regards,
Sunny

----------

If the Answer is helpful, please click "Accept Answer" and upvote it.

Note: Please follow the steps in our documentation to enable e-mail notifications if you want to receive the related email notification for this thread.

Zachariah DeGraaf 1 Reputation point

2020-12-10T16:47:49.39+00:00
I think I already answered most of your questions in my original post, but lets go again.

I looked through the basic log folders and didn't find any errors that seemed related, but I would be happy to go through others if I could be given a specific log file to look at.

I didn't make any changes to the Windows machine itself, but I did connect the NIC to a VLAN by changing settings on my hypervisor.

The one and only NIC is the virtual NIC coming from the hypervisor, so no wireless.

The NIC drivers are what is recommended for the Hypervisor I use with Windows 10. I did try changing the NIC type to see if that would make a difference, which of course used a different driver, but the same issue keeps occurring.

The only Anti-Virus I am running is Windows Defender, which has been causing it's fair share of other issues. Should I disable that?

Share via

Network Droping every 2 hours

2 answers

Your answer