[PowerShell Script] Troubleshooting for Port Exhaustion Using NetStat
Problem Description:
Applications that use a great deal of TCP network activity may use all of the possible port numbers -- especially if they are very “chatty”. By default, when an application closes a TCP connection, the port number used cannot be reused for the same IP address for another four minutes (TcpTimedWaitDelay). Also, by default, the possible port number is limited to a maximum of 5000 (MaxUserPort). Therefore, very chatty applications may use up all of the possible port numbers under very high loads – this is often called port exhaustion or socket burnout.
Troubleshooting:
To check for port exhaustion is simple, but tedious because NetStat –an will display all of the IP addresses, their port numbers and their statuses; however, it doesn’t count them!
One of my teammates, Frank Taglianetti (aka Tag), created a cool PowerShell script that counts the number of ports with the TIME_WAIT status, the percentage of used ports in TIME_WAIT, the total number of used ports, and the percentage of port numbers used.
The output is sorted by the number of ports used, in descending order so that you can see the IP address that is most likely exhausted or near exhaustion.
Compared to NetStat this script is easier to use and the results are easier to visualize.
Here is an example:
.\count-ports.ps1
[F:\Downloads\Tools\Scripts\NetStat]PS:3>.\Count-Ports.ps1 10/9/2010 6:31:06 PM IPAddress PortsWaiting %Waiting PortsUsed %Used --------- ------------ -------- --------- ----- 65.55.149.121 1 33.33 % 3 0.06 % 184.85.110.98 1 33.33 % 3 0.06 % 96.17.108.146 1 100.00 % 1 0.02 % 138.108.14.10 1 50.00 % 2 0.04 % 65.54.95.216 1 11.11 % 9 0.18 % 65.54.95.86 1 50.00 % 2 0.04 % … |
Here is the source code for the Count-Ports.ps1:
## Author: frank.taglianetti@microsoft.com
## Displays port counts per IP address
## Parameters = none
## Modified 12/6/2011 to include Windows Vista or later
## TCP Parameters documented in https://support.microsoft.com/kb/953230
function MaxNumOfTcpPorts #helper function to retrive number of ports per address
{
param
(
[parameter(Mandatory=$true)]
$tcpParams
)
# Returns the maximum number of ports per TCP address
# Check for Windows Vista and later
$IsVistaOrLater = Get-WmiObject -Class Win32_OperatingSystem | %{($_.Version -match "6\.\d+")}
if($isVistaOrLater)
{
# Use netsh to retrieve the number of ports and parse out the string of numbers after "Number of Ports : "
$maxPorts = netsh int ip show dynamicport tcp |
Select-String -Pattern "Number of Ports : (\d*)"|
%{$_.matches[0].Groups[1].Value}
# Convert string to integer
$maxPorts = [int32]::Parse($maxPorts)
# modify the PSCustomObject to simulate the MaxUserPort value for printout
Add-Member -InputObject $tcpParams -MemberType NoteProperty -Name MaxUserPort -Value $maxPorts
}
else # this is Windows XP or older
{
# check of emphermal ports modified in registry
$maxPorts = $($tcpParams | Select-Object MaxUserPort).MaxUserPort
if($maxPorts -eq $null)
{
$maxPorts = 5000 - 1kb #Windows Default range is from 1025 to 5000 inclusive
Add-Member -InputObject $tcpParams -MemberType NoteProperty -Name MaxUserPort -Value $maxPorts
}
}
return $maxPorts
}
function New-Port # helper function to track number of ports per IP address
{
Param
(
[string] $IPAddress = [String]::EmptyString,
[int32] $PortsWaiting = 0,
[int32] $MaxUserPort = 3976
)
$newPort = New-Object PSObject
Add-Member -InputObject $newPort -MemberType NoteProperty -Name IPAddress -Value $IPAddress
Add-Member -InputObject $newPort -MemberType NoteProperty -Name PortsUsed -Value 1
Add-Member -InputObject $newPort -MemberType ScriptProperty -Name PercentUsed -Value {$this.PortsUsed / $this.MaxUserPort}
Add-Member -InputObject $newPort -MemberType NoteProperty -Name PortsWaiting -Value $portsWaiting
Add-Member -InputObject $newPort -MemberType ScriptProperty -Name PercentWaiting -Value {$this.PortsWaiting / [Math]::Max(1,$this.PortsUsed)}
Add-Member -InputObject $newPort -MemberType NoteProperty -Name MaxUserPort -Value $maxUserPort
return $newPort
}
######################### Beginning of the main routine ##########################
# Store MaxUserPort for percentage used calculations
$tcpParams = Get-ItemProperty -Path HKLM:\SYSTEM\CurrentControlSet\services\Tcpip\Parameters
$maxPorts = MaxNumOfTcpPorts($tcpParams) # call function to return max # ports as per OS version
$tcpTimedWaitDelay = $($tcpParams | Select-Object TcpTimedWaitDelay).TcpTimedWaitDelay
if($tcpTimedWaitDelay -eq $Null) #Value wasn't configured in registry
{
$tcpTimedWaitDelay = 240 #Default Value if registry value doesn't exist
Add-Member -InputObject $tcpParams -MemberType NoteProperty -Name TcpTimedWaitDelay -Value $tcpTimedWaitDelay #fake reg value for output
}
# display current date and time
Write-Host -Object $(Get-Date)
# Display the MaxUserPort and TcpTimedWaitDelay settings in the registry if available
$tcpParams | Format-List MaxUserPort,TcpTimedWaitDelay
# collection of IP Address and port counts
[System.Collections.HashTable] $ports = New-Object System.Collections.HashTable
[int32] $intWait = 0
netstat -an |
Select-String "TCP\s+.+\:.+\s+(.+)\:(\d+)\s+(\w+)" |
ForEach-Object {
$key = $_.matches[0].Groups[1].value # use the IP address as hash key
$Status = $_.matches[0].Groups[3].value # Last group contains port status
if("TIME_WAIT" -like $Status)
{
$intWait = 1 # incr count
}
else
{
$intWait = 0 # don't incr count
}
if(-not $ports.ContainsKey($key)) #IP Address not yet counted
{
$port = New-Port -IPAddress $key -PortsWaiting $intWait -MaxUserPort $maxPorts #intialize new tracking object
$ports.Add($key,$port) #Add the tracking object to hashtable
}
else #otherwise a tracking object exists for this IP
{
$port = $ports[$key] #retrieve the tracking object
$port.PortsUsed ++ # increment the port count (PortsUsed)
$port.PortsWaiting += $intWait # increment PortsWaiting if status is TIME_WAIT
}
}
# Format-Table -InputObject $ports.Values -auto
$ports.Values |
Sort-Object -Property PortsUsed, PortsWaiting -Descending |
Format-Table -Property IPAddress,PortsWaiting,
@{Name='%Waiting';Expression ={"{0:P}" -f $_.PercentWaiting};Alignment="Right"},
PortsUsed,
@{Name='%Used';Expression ={"{0:P}" -f $_.PercentUsed}; Alignment="Right"} -Auto
Remove-Variable -Name "ports"
Comments
Anonymous
October 11, 2010
It's cool script. Our network team handle this issue a lot. I will share it our team.Anonymous
April 05, 2011
The default user ports available on Vista and higher is 16,363 and not 5,000. support.microsoft.com/.../929851Anonymous
April 02, 2012
This script was update! (Thanks Tag!)Anonymous
July 25, 2012
Cool Script! I noticed the TCPTimedWaitDelay is defaulted to 240 if not present, but I believe in 2008 and higher it is now defaulted to 120.Anonymous
April 18, 2013
Hi, great stuff. Question, how do I get the output of this into an eventlog?