I am still traveling to undergo trainings for my new role, but will eventually settle down in Hyderabad (July 2008?).

A windows service application (originally developed with VS2003/.NET Framework 1.1) is designed to make several HTTP(s) requests in multiple threads simultaneously. With upgrading to VS2008 and recompiling the code, the app started seeing several requests failing with "The operation has timed out" exceptions.

Taking a System.Net trace (https://msdn2.microsoft.com/en-us/library/ty48b824(VS.80).aspx) showed several threads failing as follows.

System.Net Verbose: 0 : [6044] WebRequest::Create(https://www.someserver.com/)

    ProcessId=9616

    DateTime=2008-02-08T21:45:52.3396033Z

System.Net Verbose: 0 : [6044] HttpWebRequest#38203409::HttpWebRequest(https://www.someserver.com/)

    ProcessId=9616

    DateTime=2008-02-08T21:45:52.3396033Z

System.Net Verbose: 0 : [6044] Exiting HttpWebRequest#38203409::HttpWebRequest()

    ProcessId=9616

    DateTime=2008-02-08T21:45:52.3396033Z

System.Net Verbose: 0 : [6044] Exiting WebRequest::Create()   -> HttpWebRequest#38203409

    ProcessId=9616

    DateTime=2008-02-08T21:45:52.3396033Z

System.Net Verbose: 0 : [6044] HttpWebRequest#38203409::GetResponse()

    ProcessId=9616

    DateTime=2008-02-08T21:45:52.4021029Z

System.Net Information: 0 : [6044] Associating HttpWebRequest#38203409 with ServicePoint#58391824

    ProcessId=9616

    DateTime=2008-02-08T21:46:12.4957243Z

System.Net Error: 0 : [6044] Exception in the HttpWebRequest#38203409::EndGetResponse - The operation has timed out

    ProcessId=9616

    DateTime=2008-02-08T21:46:12.4957243Z

From the above we can see that the request failed almost immediately with the exception. Taking a network trace I confirmed that the specific request above was not sent over the wire.

Checking the trace at the beginning I could see that the application started making numerous requests. However before these requests were sent on wire, .NET Framework was trying to detect auto proxy for the urls submitted. This took about 5-7 seconds, by which time there were numerous requests in queue. The timeout set for each request was 20 seconds, however most of them started failing as shown above.

Since auto proxy detection was added only from 2.0 and above, the application did not see the same problem with 1.1 .NET Framework.

In this case, the following were the possible solutions.

1. Disabling auto proxy and configure static proxy in the app.config for the application.
2. Change code to wait for the  the first request to complete (including auto proxy). Then start sending the requests from the application.

In our case, the issue was resolved by disabling auto proxy and setting static proxy in app.config. More information on doing this is available at https://msdn2.microsoft.com/en-us/library/sa91de1e(VS.80).aspx

A response accumulating web filter (such as WebResponseModifier with the ISA 2004 SDK), does the following.

1. Accumulates resoponse to client in SF_NOTIFY_SEND_RAW_DATA notification.
2. In SF_NOTIFY_END_OF_REQUEST notification, the filter writes modified or unmodified response to client using WriteClient.
3. WriteClient takes a long time to complete. This is often seen when WriteClient is used to write HTTP status line and headers.

Note: MSDN docs on WriteClient confirm that web filters may run into performance issues with WriteClient. Please refer the following for more information.

WriteClient: https://msdn.microsoft.com/library/default.asp?url=/library/en-us/isasdk/isa/writeclient.asp
Best Practices: https://msdn.microsoft.com/library/en-us/isasdk/isa/best_practices_for_web_filters.asp?frame=true

In IIS ISAPI, it is recommended NOT to use WriteClient for writing status line and headers but to use ServerSupportFunction.  This is because IIS does not detect that the response to client includes headers (which may result in IIS not correcting / erroring out if the response HTTP message is not formatted correct), if WriteClient was used.

In a similar way, the above ISA WriteClient performance issue can be resolved by using ServerSupportFunction instead. This of course adds additional couple of lines of code to your solution, however this is a cleaner and optimal way of writing response to client.

More info on ServerSupportFunction in ISA SDK is available at
https://msdn.microsoft.com/library/en-us/isasdk/isa/serversupportfunction.asp?frame=true

Example code snippet:

char szHeaders [1024] = "Cache-Control: no-cache, no-store\r\nContent-Type: text/html; charset=utf-8\r\nContent-Length: 228\r\nServer: Microsoft-IIS/6.0\r\n\r\n";

pfc->ServerSupportFunction(pfc, SF_REQ_SEND_RESPONSE_HEADER, (PVOID) "200 OK",  (ULONG_PTR) szHeaders, NULL);

As you can see szHeaders is for adding any additional headers to the response.

Note: Do not add \r\n at the end of 200 OK (status).

915599 You receive one or more error messages when you try to make an HTTP request in an application that is built on the .NET Framework 1.1 Service Pack 1
https://support.microsoft.com/default.aspx?scid=kb;EN-US;915599

********* Content below will not be updated for any new/additional info, but the KB article will be ****** So please use the KB article **********

This posting is provided "AS IS" with no warranties, and confers no rights. You assume all risk for your use. © 2003 Microsoft Corporation. All rights reserved.

Note: This discussion applies to .Net framework 1.1 and .Net framework 1.1 SP1 only.

The above exception is usually reported by System.Net.HttpWebRequest class (which may bubble up to System.Net.WebClient or System.Web.Services.Protocols.SoapHttpClientProtocol).

Working with Microsoft customers, I know how they hate the above message. But this error is actually very helpful in determining what went wrong with the underlying connection, during HTTP request/response communication sequence between client and server processes.

Types of 'Underlying connection closed' Errors:

We typically see one of the following. The second part of the error indicates what may be the cause.

"The underlying connection was closed: Unable to connect to the remote server."
--Seen when the webserver to which the HTTP request was sent, was unreachable. Usually caused by incorrect proxy configuration and/or by dns issues.

"The underlying connection was closed: An unexpected error occurred on a receive."
--Seen when the client had sent the request in its entirety and got a TCP ACK-FIN or RST from server to close the connection, without a response from server.

"The underlying connection was closed: An unexpected error occurred on a Send."
--Seen while the client is sending the request the underlying TCP connection was closed.

"The underlying connection was closed:  The server committed an HTTP protocol violation."
--Seen when the HTTP response from the server was deemed incorrect or not conforming to the guidelines specified in HTTP 1.1 RFC (https://www.w3.org/Protocols/rfc2616/rfc2616.html). May occur if the .Net client considers the response to be a security threat, as well.

"The underlying connection was closed: Could not establish secure channel for SSL/TLS."
--Seen when the client could not successfully establish SSL/TLS channel, for HTTPS request. Usually caused by invalid/expired client or server SSL certificates.

"The operation has timed-out"
--Seen when the client timeout value expired before the server response came in. Though this is not an underlying connection closed error, the following suggestions help in addressing this.

Please refer https://msdn.microsoft.com/library/default.asp?url=/library/en-us/cpref/html/frlrfsystemnetwebexceptionstatusclasstopic.asp for determining the status when getting a WebException.

The following are also very helpful in effectively resolving or working around 'Underlying connection closed' errors.

1. Ensure your .Net framework is up to date:
==================================

Typically applying SP1 for .Net framework 1.1, which is included in Windows 2003 Server SP1, should resolve most known bugs/issues with incorrect TCP behaviour. If your application is using SSL for the HTTP request, I strongly recommend applying SP1 for .Net framework (or on a Windows 2003 Server, you may apply SP1 for the OS).

Once you apply, confirm that SP1 for .Net framework is correctly installed. To do this, you can check the registry key as per the following KB article.
318785 Determine whether service packs are installed on the .NET Framework
https://support.microsoft.com/?id=318785

Doing this may address:
--------------------------------
All the above exceptions

2. Ensure that HttpWebRequest settings are appropriate for the requests issued:
===============================================================
Note that the default connections recommended for a HTTP 1.1 client is 2 and that is the default with HttpWebRequest. If your application requires more than two concurrent connections then you will have to increase the number of concurrent connections HttpWebRequest can use. If
   (a) number of connections is not increased,
   (b) both concurrent connections are busy and
   (c) if there are pending requests which do not get a chance to use a connection,
then the pending requests may get a "The operation has timed-out" exception.

Note: If you are making a request from ASP.Net code, since the ASP.Net application is a service and may have to have more than 2 connections for handling incoming requests, increasing concurrent connections for HttpWebRequest is recommended.

Note: The webservice reference proxy code, generated by Visual Studio WSDL.exe tool (which is used when you add a webreference to a C# or VB.Net project) uses System.Net.HttpWebRequest to make the HTTP requests. So if you are seeing underlying connection closed errors on a webservice call, you will have to confirm that your webservice client code/configuration is tuned as for System.Net.HttpWebRequest.

To increase number of concurrent connections, increase ServicePointManager.DefaultConnectionLimit property starting at 12 per processor as per the following MSDN article:
https://msdn.microsoft.com/library/default.asp?url=/library/en-us/cpref/html/frlrfSystemNetServicePointManagerClassDefaultConnectionLimitTopic.asp

Note: Please ensure that any change in default values for ServicePointManager properties is done before any HTTP request is sent out.

If you want to set it in a config file, and not through code, use the <connectionManagement> Element as per the following MSDN article:
https://msdn.microsoft.com/library/default.asp?url=/library/en-us/cpgenref/html/gngrfconnectionmanagementelement.asp
The <connectionManagement> element defines the maximum number of connections to a server or group of servers.

Doing this may address:
---------------------------------
"The operation has timed-out" exception

3. KeepAlive connection issues:
=========================
We could see the problem if the Keep Alive connection is not correctly handled by .Net client process or the WebServer.
Note: KeepAlive is required for NTLM and Kerberos authentication. So if you are NOT using NTLM or Kerberos as authentication and NOT require Keep Alive for the connection, you can try setting KeepAlive to false.

For setting Timeout in HttpWebRequest object: https://msdn.microsoft.com/library/default.asp?url=/library/en-us/cpref/html/frlrfsystemnethttpwebrequestclasstimeouttopic.asp
For setting Timeout from a webservice client:
   1. Create a new class by inheriting from the generated proxy class.
   2. Add the following method to override GetWebRequest method to access the underlying HttpWebRequest object.
   C#:
    protected override WebRequest GetWebRequest(Uri uri)
    {
        HttpWebRequest webRequest = (HttpWebRequest) base.GetWebRequest(uri);
        //Setting KeepAlive to false
        webRequest.KeepAlive = false;
        return webRequest;
    }

   VB.Net:
        Protected Overrides Function GetWebRequest(ByVal uri As Uri) As System.Net.WebRequest
            Dim webRequest As System.Net.HttpWebRequest
            webRequest = CType(MyBase.GetWebRequest(uri), System.Net.HttpWebRequest)
           'Setting KeepAlive to false
            webRequest.KeepAlive = False
            GetWebRequest = webRequest
        End Function

Doing this may address:
----------------------------------
"The underlying connection was closed: An unexpected error occurred on a Send."  and "The underlying connection was closed: An unexpected error occurred on a receive." exceptions

4. For closing a KeepAlive connection before server times out:
===================================================
If the server KeepAlive connection timeout is close to the client KeepAlive connection timeout, and if the client is sending a request on the connection almost at the end of timeout duration, then we may see "The underlying connection was closed: An unexpected error occurred on a Send."  and "The underlying connection was closed: An unexpected error occurred on a receive." errors.

For this problem, we can try reducing the duration at which the client will actively purge the existing unused connection through ServicePointManager.MaxServicePointIdleTime Property.
Please refer https://msdn.microsoft.com/library/default.asp?url=/library/en-us/cpref/html/frlrfsystemnetservicepointmanagerclassmaxservicepointidletimetopic.asp.

Please try setting this value less than the server KeepAlive connection timeout.

Note: Please ensure that any change in default values for ServicePointManager properties is done before any HTTP request is sent out.

Doing this may address:
---------------------------------
"The underlying connection was closed: An unexpected error occurred on a Send."  and "The underlying connection was closed: An unexpected error occurred on a receive." exceptions
 

5. Setting ASP.Net executionTimeout if client code is in ASP.Net:
==================================================
If your client code is running in ASP.Net, please make sure that your timeouts are implemented as specified below. The ASP.Net process executionTimeout and the webservice client call timeout should be set as per the following article.

904262 The request that is sent by the HttpWebRequest class may stop responding
https://support.microsoft.com/?id=904262

Ensure that your ASP.Net webservice client timeout MUST be higher than the maximum time taken by the webservice to respond for longest running request.

Doing this may address:
--------------------------------
"The operation has timed-out"  and "The underlying connection was closed: An unexpected error occurred on a receive." exceptions

6. Ensure proxy settings are correct:
===========================

You can use the static method WebProxy.GetDefaultProxy as per https://msdn.microsoft.com/library/default.asp?url=/library/en-us/cpref/html/frlrfsystemnetwebproxyclassgetdefaultproxytopic.asp, or you can use https://msdn.microsoft.com/library/default.asp?url=/library/en-us/cpgenref/html/gngrfproxyelement.asp for defining <proxy> element in .config files.

If your environment is configured for autoproxy, refer the following.
873199 How to use autoproxy in managed code
https://support.microsoft.com/?id=873199

Doing this may address:
--------------------------------
"The underlying connection was closed: Unable to connect to the remote server." exception

7. Ensure that the server (where client code is running) does not have protocol binding limitations issue:
================================================================================

826757 FIX: Socket Initialization Does Not Succeed if Your Computer Has More
https://support.microsoft.com/?id=826757

This fix is included in .Net framework 1.1 SP1 and also in Windows 2003 SP1. If this is the cause, any System.Net socket application will fail.

Doing this may address:
---------------------------------
"The operation has timed-out"  and "The underlying connection was closed: Unable to connect to the remote server." exceptions

8. Problems in establishing SSL/TLS channels:
====================================
First verify that the server and client (if any) SSL certificates are valid, not expired and can be used from browser without any problems.

The following explain how to use SSL with client certificates with .Net framework.

901183 How to call a Web service by using a client certificate for
https://support.microsoft.com/?id=901183

https://msdn.microsoft.com/library/default.asp?url=/library/en-us/dnnetsec/html/SecNetHT13.asp

https://msdn.microsoft.com/library/en-us/dnnetsec/html/SecNetHT14.asp?frame=true

895971 How to send a client certificate by using the HttpWebRequest and
https://support.microsoft.com/?id=895971

If the problem is caused by server SSL certificate (expired, untrusted CA or common name mismatch) and if the server is trusted, the errors can be ignored by using ICertificatePolicy.CheckValidationResult method (https://msdn.microsoft.com/library/default.asp?url=/library/en-us/cpref/html/frlrfSystemNetICertificatePolicyClassCheckValidationResultTopic.asp). Please note that ignoring SSL certificate warnings in code is a possible security threat, unless the server is completely trusted.

Doing this may address:
----------------------------------
"The underlying connection was closed: Could not establish secure channel for SSL/TLS." exception

9. Server response not conforming to HTTP RFC:
======================================
.Net framework 1.1 HTTP clients strictly follow HTTP 1.1 RFC guidelines in https://www.w3.org/Protocols/rfc2616/rfc2616.html. If the server response to client does not fully comply with HTTP 1.1 RFC, the client will close the connection with the "The underlying connection was closed:  The server committed an HTTP protocol violation." exception. Usually this happens with incorrect headers and header delimiters (https://www.w3.org/Protocols/rfc2616/rfc2616-sec6.html#sec6). The above exception is often seen after applying SP1 for .Net framework 1.1.

Any response not adhering with HTTP 1.1 RFC guideline, is considered as a security threat by .Net framework client, as allowing malformed headers may expose the client for "HTTP response splitting" attacks.

If the server is trusted completely, then the above exception can be worked around by following the KB article below, at the risk of compromising security. Please note that the best course of action for this exception would be to correct the server side application/webserver to send correct responses as per HTTP 1.1 RFC.

892467 HttpWebRequest.GetResponse() returns "HTTP protocol violation" error
https://support.microsoft.com/?id=892467

Doing this may address:
-----------------------------------
"The underlying connection was closed:  The server committed an HTTP protocol violation." exception

This is done. You can read the article here https://blogs.msdn.com/engelsr/articles/497902.aspx