Network Interface – Output Queue Length values extremely high?
Applies to:
Windows Server 2003 RTM
Windows Server 2003 Service Pack 1
Windows Server 2003 Service Pack 2
Windows Server 2003 R2 Service Pack 1
Windows Server 2003 R2 Service Pack 2
Problem description:
In Perfmon, Network Interface object, Output Queue Length counter shows high.
What is Network Output Queue Length?
“Output Queue Length is the length of the output packet queue (in packets). If this is longer than two, there are delays and the bottleneck should be found and eliminated, if possible. Since the requests are queued by the Network Driver Interface Specification (NDIS) in this implementation, this will always be 0.”
What are valid values?
0-5
What are invalid values?
Large values negative or greater than 1000 are bogus, ignore.
Cause:
1. If the number is abnormally high on a consistent basis.
2. Outdated network card driver, though such a problem normally manifests itself in other ways than just an increased output queue length statistic.
3. Receive Side Scaling (RSS) is disabled, if you went to Windows Update, the likely hood that you had RSS disabled is 100%.
948496 Make sure that you have the following updates:An update to turn off default SNP features is available for Windows Server 2003-based and Small Business Server 2003-based computers
https://support.microsoft.com/kb/948496
If you have this disabled, you will run into:
892100 The processor load is not distributed across multiple processors on a computer that is running Windows Server 2003, Windows 2000 Server, or Windows NT 4.0
https://support.microsoft.com/kb/892100
4. Update the network stack for Windows Server 2003.
5. You should look for a problem with a network card or some other piece of network hardware.
6. There is too much load on the server.
Resolution:
In order of importance from our experience:
1. Ignore if the value is over 1000.
2. Upgrade the NIC driver and firmware
Upgrade the NIC teaming software (or Break the NIC teaming)
3. Re-enable RSS
•HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters
•Right-click EnableRSS, and then click Modify.
•In the Value data box, type 0, and then click OK.
Note: it requires a reboot for the change to occur
4. Update the following files:
•tcpip.sys
950224 A Scalable Networking Pack (SNP) hotfix rollup package is available for Windows Server 2003
https://support.microsoft.com/?id=950224
In https://www.bing.com, use the following keywords:
hotfix "Server 2003" tcpip.sys site:support.microsoft.com/kb/
•afd.sys
951162 After you enable the dynamic backlog settings on a Windows Server 2003-based computer, the System Handle count and the nonpaged pool usage may increase significantly
https://support.microsoft.com/kb/951162
In https://www.bing.com, use the following keywords:
hotfix "Server 2003" afd.sys site:support.microsoft.com/kb/
•ndis.sys
972071 Applications that use the UDP protocol may encounter poor performance on a computer that is running Windows Server 2003
https://support.microsoft.com/kb/972071
In https://www.bing.com, use the following keywords:
hotfix "Server 2003" afd.sys site:support.microsoft.com/kb/
•tdi.sys, no updates as of 7/6/2010
•wshtcpip.dll, no updates as of 7/6/2010.
5. For testing purpose, replace the Network Card with a different manufacturer Network Card that is a server class.
6. If NIC is does not support the maximum bandwidth of your network, upgrade the NIC to support a faster bandwidth.
An application or multiple applications are causing sudden elevated network usage patterns.
Reduce the load on the server. You can do this by moving the load to a server that is utilized less. If it is required, add an additional server to your environment.
More information:
-------------------
These are the other Network Interface counters that you should be analyzing in conjuction:
•Network Interface: Bytes Received/sec
•Network Interface: Bytes Sent/sec
•Network Interface: Bytes Total/sec
•Network Interface: Current Bandwidth