I wanted to publish this article for people that might be encountering a port exhaustion scenario with their servers. With the current existing tools, there is no nice and easy way to determine if your machine might be encountering a port exhaustion scenario and if so, what process is consuming the most ephemeral ports. Hopefully with this information, you should be able to quickly identify if you might be encountering a port exhaustion scenario with your servers and which process is responsible.

Symptoms of Port Exhaustion

System.Net.WebException: The underlying connection was closed: An unexpected error occurred on a send.

- or -

System.Net.WebException: Unable to connect to the remote server

Identify port exhaustion using ETL Tracing

• On the affected machine, currently in state, open administrator command prompt and run the following command:
  • Netsh trace start scenario=netconnection capture=yes
• Attempt to establish an outbound connection and reproduce the issue
• Stop the trace using:
  • Netsh trace stop
• Now that you have the trace captured from the machine, download and install network monitor 3.4 and load the trace.
• Open the ETL trace in netmon, and then use the following display filter to identify any 'STATUS_TOO_MANY_ADDRESSES' which would be an indicator of port exhaustion

• Wscore_MicrosoftWindowsWinsockAFD.AFD_EVENT_BIND.Status.LENTStatus.Code == 0x209

Ok so cool, now you know you are running out of ephemeral ports on your machine and that is the reason you can no longer establish outgoing connections. Now what?

Identify the Process(es) consuming ports using Process Explorer

The next step then at this point would be to identify the process that is consuming all the ephemeral ports. The easiest way to do this is to download and use process explorer from sysinternals.

• Download the Process Explorer tool and extract the procexp.exe onto the affected machine that is currently in state
• Launch Process Explorer as Administrator
  
  • Failing to launch as Administrator w/ UAC enabled will prevent the services from loading
• Navigate to View and select 'Show Lower Pane'
• Navigate to View > Select Columns and within the Process Performance tab, checkmark 'Handle Count'
• Sort the processes by Handle Count to locate the services consuming the most handles on the OS
• Within the Lower Pane, sort the results by Name
• At this point, you will then want to examine the processes to locate \device\afd which is an indication that the process has an ephemeral port opened. Since we running into a ephemeral port exhaustion, you will want to locate a process where you see thousands of instances of \device\afd defined.

In my scenario above in the screenshot, I don't have port exhaustion but I wanted to make sure I provided an image for clarification purposes of what you are looking for. Once you have identified the process that is consuming the ephemeral ports, you will want to follow up with the necessary company that owns that component to have them investigate further to determine if there is a issue with the code or if you are encountering another environment factor.

Let me know if this is helpful!

Related Articles:

Process Explorer v16.04

https://technet.microsoft.com/en-us/sysinternals/bb896653.aspx

Network Monitor 3.4

https://www.microsoft.com/en-us/download/details.aspx?id=4865

First things first, ensure you have read https://msdn.microsoft.com/en-us/library/windowsazure/dn195495.aspx to ensure you have deleted all the necessary objects that are utilizing the VNET. So let's say you have done what was instructed in the article and it's still not allowing you to delete the VNET! One thing to check for is to ensure the Point-to-Site VPN connection checkbox is unchecked as this will prevent the VNET from being deleted.

After you uncheck this box and then Save the changes, you should now be able to go and delete the VNET as long as all other resources have been removed from the VNET as described in the MSDN Azure article mentioned above.

(&(objectCategory=person)(objectClass=user)(!(BIT_AND:
(userAccountControl)&2))(|(name=adam*)(displayName=adam*)(cn=adam*)(mail=adam*)(sn=adam*)(SamAccountName=adam*)(proxyAddresses=SMTP:adam)(proxyAddresses=sip:adam)))

 
(&(objectCategory=group)(BIT_AND:
(groupType)&2147483648)(|(name=adam*)(displayname=adam*)(cn=adam*)(SamAccountName=adam*)))

However in SharePoint 2013, even though we still use the application pool account and still using LDAP request, our filter has changed.

 
(&(objectCategory=person)(|(anr=adam*)(SamAccountName=adam*)))

(&(objectCategory=group)(BIT_AND:
(groupType)&2147483648)(|(anr=adam*)(SamAccountName=adam*)))

So per our documentation at https://technet.microsoft.com/en-us/library/cc978014.aspx the ANR attribute stands for Ambiguous Name Resolution and is a set of attributes that Active Directory evaluates when it receives an LDAP Search Request. By default, the following attributes are associated with ANR.

• givenName (first name)
• sn (surname, or last name)
• displayName (the name given the object when it is created)
• RDN (the relative distinguished name of the object)
• legacyExchangeDN (for enterprises that have upgraded a Microsoft® Exchange installation to a later version of Exchange that is synchronized with Active Directory, the distinguished name of the old Exchange mailbox that corresponds to the user in Active Directory)
• physicalDeliveryOfficeName (for example, Building A, Suite 1234)
• proxyAddresses (the collection of e-mail addresses over all e-mail address spaces that the Exchange server knows about)

So this definitely can cause some issues in SharePoint 2013 if your SharePoint 2007 or SharePoint 2010 environments were previously resolving usernames based on their mail attribute in AD. In order to fix this, you need to perform the following:

1. On the DC, open RUN and type REGSVR32 SCHMMGMT.DLL to register the Active Directory Schema Snap-in
2. Open MMC console via RUN command
3. Navigate to File > Add/Remove Snapin and load the Active Directory Schema MMC
4. Expand Active Directory Schema [ServerName.Domain.Com] and select Attributes
5. Locate the mail attribute and right-click properties
6. Check 'Ambiguous Name Resolution (ANR)' and select Apply. Select OK
7. Right-Click on 'Active Directory Schema[ServerName.Domain.Com] and select 'Reload the Schema'

Now you should be able to perform a search for value stored in the mail attribute in SharePoint 2013 and return results! The cool thing about this type of configuration is that you can easily extend your Active Directory schema using the same steps above to allow SharePoint to query other attributes in your domain environment without having to write your own custom SharePoint queries as previous with older versions of SharePoint.

Hope this helps!

SharePoint Server is not correctly configured to connect to Exchange Server

One or more pre-requisites for connecting to Exchange Server could not be found. Please contact your system administrator to resolve this issue and then try again.

Correlation ID: GUID, Error Code 1

I checked the Event Viewer logs and the ULS logs matching up with the correlation ID thrown on the page, however nothing stood out as a failure point. Even enabling verbose logging did not point me in the right direction. After digging, the issue I found was I had accidentally installed the MSI package by double-clicking on it after I downloaded it and went through the install method. This will not work as need to run the command line as indicated in Configure site mailboxes in SharePoint 2013 documentation to properly install MSI package and ensure it is added to the GAC.

msiexec /i EwsManagedApi.msi addlocal="ExchangeWebServicesApi_Feature,ExchangeWebServicesApi_Gac"

Just thought I would throw this out there in case anyone encounters this. If you do, just double-click the EwsManagedApi.msi file again and select the option to remove it from the computer, and then run the above command to re-install it properly. After performing IISRESET then you should be able access the Site Mailbox app without receiving Error Code 1.

I was troubleshooting my SharePoint 2013 Farm and clearing out the error messages flooding my Application Event Viewer with following message:

"The Execute method of job definition Microsoft.Office.Server.Search.Monitoring.HealthStatUpdateJobDefinition (ID 546903e6-ab54-43af-8ea1-0e30c89a75e7) threw an exception. More information is included below.  Microsoft SharePoint Foundation Usage service application not found."

Checking ULS Logs, I did not see much more detail than above,

01/25/2013 09:57:07.54 OWSTIMER.EXE (0x17C0) 0x0888 SharePoint Foundation Timer 6398 Critical The Execute method of job definition Microsoft.Office.Server.Search.Monitoring.HealthStatUpdateJobDefinition (ID 546903e6-ab54-43af-8ea1-0e30c89a75e7) threw an exception. More information is included below.  Microsoft SharePoint Foundation Usage service application not found. a149f89b-ca65-c06e-cf6c-54075dddd87e

So I figured this would be a Service Application that was supposed to be provisioned under Application Management > Manage Service Applications, however this was not the case. Doing some digging, I found that this Usage Service Application is actually provisioned from another location.

In order to fix this, you will want to navigate to Central Admin > Monitoring and select Configure Usage and Health Data Collection under Reporting. Configure the settings how you want and then select OK. Once this has been completed, when you navigate to Manage Service Applications, you will see a "WSS_UsageApplication" service application listed.

So playing around with my sites the other day, I noticed an issue where in SharePoint 2013 publishing site, my URL was using the old navigation structure as previous versions of SharePoint and not the clean looking navigation that is now available in SharePoint 2013 causing my URL to look like https://www.contoso.com/pages/default.aspx instead of https://www.contoso.com.

So doing some poking around, I went to Site Settings > Navigation and seen I had the following settings defined.

So I attempted to switch it to the Managed Navigation option and under Structural Navigation:Term Set section, it shows my Term Sets from the Managed Metadata Service Application however nothing really to select and if I choose 'Create Term Set' it fails:

Okay, so now with that error message I assume that something is wrong with my current Managed Metadata Service Application and start looking through Event Viewer, ULS logs and checking to ensure I can access the web service. After doing a bunch of testing and even provisioning a new MMS, I located a hidden setting for the Managed Metadata Proxy that was not enabled with my configuration.

After enabling the property above, everything started working properly and my sites were then using the Managed Navigation feature that I wanted. Doing some investigation, found that if you use the SharePoint Wizard to configure the farm, this option will be automatically enabled and you never encounter this issue; however if you use the manual configuration option that many SharePoint Administrators prefer, this option WILL NOT be checked and definitely can cause some confusion.

Anyways hope this information is able to help you out!

Additional Information:

Managed Navigation - https://msdn.microsoft.com/en-us/library/jj163978(v=office.15).aspx

I wanted to get a blog published out there for people encountering issues when installing SharePoint Server packages and receive an errors similar to the below:

"The expected version of the product was not found"

"The detection failed. This can be due to a corrupted installation database"

 "An error occurred while running detection"

Hopefully the information below will be able to assist you and how you can self-troubleshoot and resolve the issue on your own. In my scenario, I am attempting to apply SP1 Language Pack for SharePoint and I am encountering issues.

In order to troubleshoot these issues, you will want to compile and execute the ROIScan.vbs script from https://technet.microsoft.com/en-us/library/hh221405 on the server that you are encountering the issue with installing the update. When the report generates, you will want to scan anything under the Review Items section to ensure there are no errors, in my case, I see the following:

Review Items
============
Error:   A chained product is missing which breaks the ability to maintain or uninstall this product. Config Product: Office14.OSERVER, Missing Product: {90140000-101F-040C-1000-0000000FF1CE}
Error:   A script internal error occurred. The integrity of the logged data might be affected

So from the results above, I can see that I am missing Office14.OSERVER for Product 90140000-101F-040C-1000-0000000FF1CE. Now the easiest way to determine what is exactly missing is to execute the ROIScan.vbs script on a server that I was able to patch correctly and does not have any issues to compare the Chained Packages section for each Product. If we scan the ROIscan report, we should notice that the report has a lot of information and should see a section for SharePoint Server 2010, any language packs installed as well as Project Server or Office Web Apps. In my scenario, the message is throwing an error on a missing chained product related to Office14.Oserver which indicates I need to look at the Microsoft SharePoint Server 2010 section. I have included the results below for my SharePoint 2010 chained packages to make it easier to understand.

Microsoft SharePoint Server 2010
--------------------------------                      
Chained Packages            
{90140000-1138-0000-1000-0000000FF1CE} - 14.0.6029.1000 - Microsoft Excel Mobile Viewer Components
{90140000-1015-0409-1000-0000000FF1CE} - 14.0.6029.1000 - Microsoft SharePoint Foundation 2010 1033 Lang Pack
{90140000-101F-0409-1000-0000000FF1CE} - 14.0.6029.1000 - Microsoft Office Server Proof (English) 2010
{90140000-101F-0419-1000-0000000FF1CE} - 14.0.6028.1000 - Microsoft Office Server Proof (Russian) 2010
{90140000-101F-0407-1000-0000000FF1CE} - 14.0.6028.1000 - Microsoft Office Server Proof (German) 2010
{90140000-101F-0401-1000-0000000FF1CE} - 14.0.6028.1000 - Microsoft Office Server Proof (Arabic) 2010
{90140000-1088-0000-1000-0000000FF1CE} - 14.0.6029.1000 - Microsoft Slide Library
{90140000-10FD-0409-1000-0000000FF1CE} - 14.0.6029.1000 - Microsoft Search Server 2010 English Language Pack
{90140000-10F8-0409-1000-0000000FF1CE} - 14.0.6029.1000 - Microsoft Excel Services English Language Pack
{90140000-10F1-0409-1000-0000000FF1CE} - 14.0.6029.1000 - Microsoft Visio Services English Language Pack
{90140000-10F0-0409-1000-0000000FF1CE} - 14.0.6029.1000 - PerformancePoint Services in SharePoint 1033 Language Pack
{90140000-1123-0000-1000-0000000FF1CE} - 14.0.6029.1000 - Microsoft Access Services Server
{90140000-112A-0000-1000-0000000FF1CE} - 14.0.6029.1000 - Microsoft Web Analytics Web Front End Components
{90140000-104B-0000-1000-0000000FF1CE} - 14.0.6029.1000 - Microsoft SharePoint Portal
{90140000-104C-0000-1000-0000000FF1CE} - 14.0.6029.1000 - Microsoft User Profiles
{90140000-10EE-0000-1000-0000000FF1CE} - 14.0.6029.1000 - PerformancePoint Services for SharePoint
{90140000-10EC-0000-1000-0000000FF1CE} - 14.0.6029.1000 - Microsoft Word Server
{90140000-101F-0C0A-1000-0000000FF1CE} - 14.0.6028.1000 - Microsoft Office Server Proof (Spanish) 2010
{90140000-1131-0409-1000-0000000FF1CE} - 14.0.6029.1000 - Microsoft Web Analytics English Language Pack
{90140000-10D7-0000-1000-0000000FF1CE} - 14.0.6029.1000 - Microsoft InfoPath Forms Services
{90140000-1014-0000-1000-0000000FF1CE} - 14.0.6029.1000 - Microsoft SharePoint Foundation 2010 Core
{90140000-104E-0409-1000-0000000FF1CE} - 14.0.6029.1000 - Microsoft SharePoint Portal English Language Pack
{90140000-10ED-0409-1000-0000000FF1CE} - 14.0.6029.1000 - Microsoft Word Server English Language Pack
{90140000-1080-0409-1000-0000000FF1CE} - 14.0.6029.1000 - Microsoft Shared Coms English Language Pack
{90140000-110D-0000-1000-0000000FF1CE} - 14.0.6029.1000 - Microsoft SharePoint Server 2010
{90140000-107F-0000-1000-0000000FF1CE} - 14.0.6029.1000 - Microsoft Shared Components
{90140000-1104-0409-1000-0000000FF1CE} - 14.0.6029.1000 - Microsoft Slide Library English Language Pack
{90140000-1103-0409-1000-0000000FF1CE} - 14.0.6029.1000 - Microsoft Document Lifecycle Components English Language Pack
{90140000-1124-0409-1000-0000000FF1CE} - 14.0.6029.1000 - Microsoft Access Services English Language Pack
{90140000-10FB-0000-1000-0000000FF1CE} - 14.0.6029.1000 - Microsoft Search Server 2010 Core
{90140000-10F7-0000-1000-0000000FF1CE} - 14.0.6029.1000 - Microsoft Document Lifecycle Components
{90140000-10F6-0000-1000-0000000FF1CE} - 14.0.6029.1000 - Microsoft Excel Services Components
{90140000-10F3-0000-1000-0000000FF1CE} - 14.0.6029.1000 - Microsoft Visio Services Web Front End Components
{90140000-10D8-0409-1000-0000000FF1CE} - 14.0.6029.1000 - Microsoft InfoPath Form Services English Language Pack

After comparing with a working server where the patches applied successfully, I see that I am missing the 'Microsoft Office Server Proof (French) 2010' chained package above. Now the easiest way to get this installed back into the environment is to use you SharePoint Installer binaries and extract them. The file I am looking for is the svrproof.msi file located in the OSRV.en\us\SvrProof.fr folder of the SharePoint Installer binaries. In order to install this file, I just execute the following command.

MSIEXEC /i "C:\SharePointInstallerFolder\OSRV.en-us\SvrProof.fr\svrproof.msi" LAUNCHEDBYSETUPEXE=1 /qb

So now when I execute the ROIScan.vbs script again, I now see the 'Microsoft Office Server Proof (French) 2010' package listed in my Chained Packages. However when I execute the package I am attempting to apply, I still receive "The expected version of the product was not found" even though my ROIScan.vbs no longer is showing any error messages and everything looks clean. This is the fun part where I now need to start looking more closely at the Chained Packages, Patch Baseline and Patchlist for each Product. So if you look at my scan above, look closely at the version numbers listed for each chained package and you will see something odd.

If you haven't noticed it yet, it’s the versions that are listed for the language chained products. The French language at this time is fine as we just installed the MSI package so it is expected it would show as 14.0.4763.1000(RTM). However Russian, German, Arabic and Spanish in the report are showing as 14.0.6028.1000. This is where my failure point is now as the package is expecting at least 14.0.6029.1000(SP1) to be installed for these chained packages.

In order to fix this issue then with pre-SP1 packages installed, I like to navigate to https://support.microsoft.com/kb/971187 and run the wizard.

Once the wizard completes, the component should be removed and you can begin the wizard again. In this scenario, I will remove the following:

Microsoft Office Server Proof (Russian) 2010
Microsoft Office Server Proof (German) 2010
Microsoft Office Server Proof (Arabic) 2010
Microsoft Office Server Proof (Spanish) 2010

Once the packages have been removed, I just need to re-install the MSI package for each language I removed. I can use the existing MSIEXEC command as previously for the French language and just update the language directory the msi package resides in.

MSIEXEC /i "C:\SharePointInstallerFolder\OSRV.en-us\SvrProof.fr\svrproof.msi" LAUNCHEDBYSETUPEXE=1 /qb

Now when I execute the ROIScan.vbs script and review the report, I see that the language chained packages are now listed at the RTM build of 14.0.4763.1000. I now just have one last step to perform and that is to get the chained packages up to SP1 build which will then allow me to apply the SP1 language pack(s). I usually prefer downloading SP1 again from Microsoft Download site as the previous SP1 package that was used may be corrupted. I can always check this by opening up the SP1 package and looking at the svrproofsp1-{Language}.xml file and checking the UpdatedVersion field if I want to double-check.

Once I have the SP1 package extracted, I just execute the following commands or double-clicking on the MSP package for each language to install it.

MSIEXEC /p "C:\SharePointSP1\svrproofsp1-fr-fr.msp" LAUNCHEDBYSETUPEXE=1 /qb

NOTE: If you are unfamiliar with extracting the MSI and MSP packages from an exe, just open a command prompt, change to directory where the .exe file exists and type "Filename.exe /extract" which will then launch a EULA screen and then ask where you wish to extract the files to.

I now at this time execute ROIScan one last time to ensure my versions are listed properly for the chained packages I was working with and if everything does, I then attempt to install the package again which should succeed at this time unless there are some other updates or such missing on the server. One thing to be aware of is if you have Office Web Apps or Project Server installed, there is a good chance those chained packages for those products are incorrect if you had issues with SharePoint Server chained packages and you will want to review the ROIScan report. If things do not look solid, then you just need to perform the same steps as listed above for each product that is effected.

**UPDATE**

Another thing I have seen happen a lot of times after patching the system correctly, is that when you launch the Product and Technologies Wizard to run an upgrade, it may throw an error similar to the following:

Error: Some farm products and patches were not detected on this or other servers. If products or patches are missing locally, you must quit this program and install the required products and patches on this server before restarting this wizard. If products or patches are missing on other servers, you must install the required products and patches on the specific servers, and you may then click the refresh button to perform the status check again.

If you do encounter this issue, and the patches it indicates are related to the German, French, Spanish, Russian or Arabic languages, then you just need to run Get-SPProduct -local which should fix this on each server. The reason you encounter this is that in the configuration database, we store a record of the patches installed in each of the servers and this can come out of sync with the registry key patches on the servers. This command will cause it to re-sync and reflect the server updates properly.