Recently I was working with a customer who was trying execute a SSIS package programmatically from an ASP.NET application. The package used the SendMail task to send emails to user. The task had three properties which were being evaluated using an expression from local SSIS variables. The values of the local variables were dynamically being passed from the ASP.NET application. Everything worked fine but sometimes the package failed from the ASP.NET application. As opposed to this, the package ran successfully every time from BIDS or from Management Studio. If I use the same code to execute the package from a console app, it ran successfully every time. So there was something different from the ASP.NET application. To understand it better, I set up a repro in my lab machine and also did some instrumentation to my code. The variable values were being updated properly but still the package failed. On Error event handler captured the following error:

Microsoft.SqlServer.Dts.Runtime.TaskHostThe result of the expression 
"@[User::MailRecipient]" on property "ToLine" cannot be written to the property. 
The expression was evaluated, but cannot be set on the property.

I also observed that I cannot access the Properties.GetValue or Properties.SetValue of the TaskHost object as described in here.
If you try this from a console app, it works fine, but breaks every time from an ASP.NET application. Getting this exception:

Attempted to access an unloaded AppDomain.
[AppDomainUnloadedException: Attempted to access an unloaded AppDomain.]
Microsoft.SqlServer.Dts.Runtime.DtsProperty.SetValue(Object o, Object value) +146
SSISWebApp._Default.Page_Load(Object sender, EventArgs e) in D:\SSISWebApp\SSISWebApp\Default.aspx.vb:73
System.Web.UI.Control.OnLoad(EventArgs e) +91
System.Web.UI.Control.LoadRecursive() +74
System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint) +2207

Surprisingly every time I made any changes in my ASP.NET code and rebuild my assembly, I landed into these errors until I recycle my app pool. Once I recycle the app pool the code executes without any error. We did some live debugging and found that it’s a bug in the SSIS object model.

What we discovered is that within the w3wp process, our DTS assemblies are stored within the Shared AppDomain whereas the actual ASP.NET application is housed in a separate AppDomain for itself.  When any change is made to the assembly in the project AppDomain, it is completely torn down due to ASP.NET AppDomain’s inability to dynamically load DLLs.  When this occurs, any subsequent attempts to access the properties on a TaskHost fails due to some vestigial reference in our code in the shared AppDomain back to the original. Recycling the AppPool resolves the issue, but any subsequent updates to the application will require another recycle. The issue occurs due to caching that our DLLs do in order to avoid doing reflection multiple times on COM components.  Cross AppDomain interactions have long been established as a weak point of SSIS due to this limitation.

Also, the issue is not limited to our SendMail Task. It would occur on any Tasks/Components since they all boil down to underlying COM objects. Users would face this behavior more on their test environments as compared to production. Because while testing we intend to change our code multiple times and rebuild our app but we seldom do that in production. If you are facing the above issue I would recommend two simple workarounds:

1. Isolate your application in a separate application pool.
2. Recycle your app pool every time you rebuild your assembly. Since the app is running on a separate app pool, it wont affect other applications. 

SELECT CustomerName.CustID, CustomerName.City, CustID.PhoneNo
FROM CustomerName INNER JOIN

CustID ON CustomerName.CustID = CustID.CustID

Notice that the table CustID has a column with the same name, i.e. CustID. As a result the query was being modified inside Query Designer:

SELECT CustomerName.CustID, CustomerName.City, dbo.CustomerName.CustID.PhoneNo
AS Expr1
FROM dbo.CustomerName INNER JOIN
dbo.CustID ON CustomerName.CustID = dbo.CustomerName.CustID.CustID

However if we open up a Query Designer and try to build the query inside it from scratch, it was able to create it. But if we provide it with a pre-written query which has a column name same as its table name, it was not able to reverse engineer the query. I tested this with the Management Studio Query Designer, and that does not have this issue. I also tested it with BIDS 2005 Query Designer and that does not have this issue either.

It was odd because the Query Designer code base was not really changed. So the only difference I can see is the provider it was using to build the Query. BIDS 2005 OLEDB Source uses a connection manager with the SQLNCLI OLEDB Provider. When we migrate the package to SSIS 2008, the provider automatically changes to SQLNCLI10. Whereas when you use the same Query Designer from SQL Management Studio, it will use .NET SqlClient Provider.

An easy workaround to avoid this would be to use the OLEDB Source in SSIS 2008 with a connection manager using the SQLNCLI Provider instead of SQLNCLI10.

Consider you have a .Net application which is connecting to a SQL Server 2005 or 2008 database which is mirrored to another instance of SQL Server using the .NET SQLClient Provider. Intermittently you may see connectivity errors from your application with the following stack trace:

Timeout expired. The timeout period elapsed prior to completion of the operation or the server is not responding.
at System.Data.SqlClient.SqlInternalConnection.OnError(SqlException exception, Boolean breakConnection)
at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning()
at System.Data.SqlClient.TdsParserStateObject.ReadSniError(TdsParserStateObject stateObj, UInt32 error)
at System.Data.SqlClient.TdsParserStateObject.ReadSni(DbAsyncResult asyncResult, TdsParserStateObject stateObj)
at System.Data.SqlClient.TdsParserStateObject.ReadNetworkPacket()
at System.Data.SqlClient.TdsParser.ConsumePreLoginHandshake(Boolean encrypt, Boolean trustServerCert, Boolean& marsCapable)
at System.Data.SqlClient.TdsParser.Connect(ServerInfo serverInfo, SqlInternalConnectionTds connHandler, Boolean ignoreSniOpenTimeout, Int64 timerExpire, Boolean encrypt, Boolean trustServerCert, Boolean integratedSecurity)
at System.Data.SqlClient.SqlInternalConnectionTds.AttemptOneLogin(ServerInfo serverInfo, String newPassword, Boolean ignoreSniOpenTimeout, TimeoutTimer timeout, SqlConnection owningObject)
at System.Data.SqlClient.SqlInternalConnectionTds.LoginWithFailover(Boolean useFailoverHost, ServerInfo primaryServerInfo, String failoverHost, String newPassword, Boolean redirectedUserInstance, SqlConnection owningObject, SqlConnectionString connectionOptions, TimeoutTimer timeout)

What is more surprising is that the timeout happens instantaneously, like within a second. Ideally you would expect to get a timeout after 15 seconds (the default connection timeout), buts that’s not the case.

If we take a Netmon trace, here is what we see:

In this example we can see the app server closes the connection close to 1 sec during the TDS: Prelogin. When the SQL Server responds back in 5 sec its too late and client will reset the connection.

As per the Connection Retry Algorithm for Mirroring, SqlClient code takes the connection timeout and multiplies it by 0.08 to determine the milliseconds to wait for first read (SniReadSync call). Now if this first SniReadSync fails due to a timeout (due to a slow response from the server), the connection is incorrectly set to a doomed state by the sqlClient .NET Provider and this causes the connection attempt to prematurely fail before the entire connection timeout expires.

Similarly if you look into the Connectivity Ring Buffer in SQL Server, you will notice that the entire login process got timed out in a short timeframe of milliseconds.

You can run the following query to get the ring buffer output from Management Studio:

SELECT CAST(record AS XML) FROM sys.dm_os_ring_buffers 
WHERE ring_buffer_type = 'RING_BUFFER_CONNECTIVITY'

But all of the above were symptoms of this issue. So how can you be sure if you are actually hitting this problem? The answer is BID tracing. I would not go into details of what is BID tracing and how to collect it. You can read about it here. All you need to trace is System.Data.SNI.1 and System.Data.1 from the application server. Here is a sample of what you will see in the trace:

<prov.DbConnectionHelper.ConnectionString_Set|API> 1#, 'Data 
Source=MySQLServer;Failover Partner=MySQLServer2; Initial Catalog= 
TestMirror;Integrated Security=SSPI;Connection Timeout=15'

enter_02 <SNIReadSync|API|SNI> 2#{SNI_Conn}, pConn: 
007D9EA8{SNI_Conn*}, ppNewPacket: 0671E268{SNI_Packet**}, timeout: 1134 

enter_03 <Np::ReadSync|API|SNI> 3#, ppNewPacket: 
0671E268{SNI_Packet**}, iTimeOut: 1134 

<SNI_Packet::SNIPacketAllocateEx2|API|SNI> pConn: 007D9EA8{SNI_Conn*}, 
IOType: 0, consumer: 0 
<SNI_Packet::SNIPacketNew|API|SNI> pConn: 007D9EA8{SNI_Conn*}, IOType: 0 
ObtainIDa 5# <SNI_Packet::SNI_Packet|ID|SNI> 0624EEF0{.} 

<SNI_Packet::SNI_Packet|SNI> 5#{SNI_Packet} created by 2#{SNI_Conn} 

<SNI_Packet::SNIPacketNew|RET|SNI> 0624EEF0{SNI_Packet*} 

<SNI_Packet::SNIPacketAllocateEx2|RET|SNI> 0624EEF0{SNI_Packet*} 

<Np::ReadSync|ERR|SNI> ProviderNum: 4{ProviderNum}, SNIError: 
11{SNIError}, NativeError: 258{WINERR} 
<Np::ReadSync|RET|SNI> 258{WINERR} 
leave_03 
<SNIReadSync|RET|SNI> 258{WINERR}, Packet: 00000000 
leave_02 
<sc.SqlError.SqlError|ERR> infoNumber=-2, 
errorState=0, errorClass=11, errorMessage='Timeout expired. The timeout period 
elapsed prior to completion of the operation or the server is not responding.', 
procedure='', lineNumber=0 

<sc.SqlInternalConnectionTds.BreakConnection|RES|CPOOL> 4#, Breaking connection. 

<prov.DbConnectionInternal.DoomThisConnection|RES|INFO|CPOOL> 4#, Dooming 
<sc.SqlInternalConnectionTds.LoginFailure|RES|CPOOL> 4#

When Connection Timeout=15 sec (default value), actual Connection Timeout calculated= .08*15 ~ 1.2 sec

From the trace you can see the SNIReadSync is setting the timeout to 1134 milliseconds. This is close to 1.2 seconds and when the server does not respond back within that time, it incorrectly dooms the connection. Truly speaking you do not actually need a mirrored database for this. All you need to mention is "Failover Partner" in your connection string. That is enough to take this connection retry algorithm code path.

While this problem should be fixed in Visual Studio 2011, there a few workarounds that can easily avoid this.

1. You can increase the connection string timeout to 150 sec. This will give the first attempt of SniReadSync enough time to connect (150* .08=12 sec)

2. Add "Min Pool Size=20" in the connection string. This will always maintain a minimum of 20 connections in the pool and there will be less need of creating new connection, thus reducing the chance of this error.

3. Improve the network performance. Most of these issues can be a result of a slow network. Update your NIC drivers to the latest firmware version. We have seen network latency when your NIC card is not compatible with certain Scalable Networking Pack settings, most commonly the TCP Chimney offload and the Receive Side Scaling(RSS) . If you are on Windows Vista SP1 or above you may also consider disabling Receive Window Auto-Tuning. If you have NIC teaming enabled, disabling it would be a good option.

I have many times faced issues where a regular domain user is able to successfully connect to SQL Server but any application running under the built in accounts are failing.

Testing connectivity with these accounts had always been challenging. In the past I used the AT command to schedule a windows task to launch a command in interactive mode. This launches cmd.exe under the SYSTEM account. Then I can test the SQL connectivity from here using osql or odbcad32.exe

https://support.microsoft.com/kb/313565

This worked fine with Windows 2003 Server or Windows XP. But starting from Vista and higher, the AT command does not launch the task in interactive mode due to security reasons.

The alternative I found was to use the PSExec Tool from SysInternals.

Run a cmd from PsExec tool and the command will launch under the SYSTEM account.

https://technet.microsoft.com/en-US/sysinternals/bb897553.aspx

Log Name: Security 
Source: Microsoft-Windows-Security-Auditing 
Date: 1/15/2011 2:52:01 PM 
Event ID: 4625 
Task Category: Logon 
Level: Information 
Keywords: Audit Failure 
User: N/A 
Computer: SQLMACHINE.corp.mydomain.com 
Description: 
An account failed to log on. 
Subject: 
Security ID: NULL SID 
Account Name: - 
Account Domain: - 
Logon ID: 0x0 
Logon Type: 3 
Account For Which Logon Failed: 
Security ID: NULL SID 
Account Name: APPSERVER$ 
Account Domain: CORP.MYDOMAIN.COM 
Failure Information: 
Failure Reason: Unknown user name or bad password. 
Status: 0xc000006e 
Sub Status: 0x0 
Process Information: 
Caller Process ID: 0x0 
Caller Process Name: - 
Network Information: 
Workstation Name: - 
Source Network Address: - 
Source Port: - 
Detailed Authentication Information: 
Logon Process: Kerberos 
Authentication Package: Kerberos 
Transited Services: - 
Package Name (NTLM only): - 
Key Length: 0

In this scenario, the app was trying to connect with the machine account. Typically when using windows authentication, if the application is running in the context of Local System or Network Service, the application will connect using the machine account. If you take a Profiler trace, the account name is shown as MachineName$. 
If we add the APPSERVER$ account to the Administrators group of the SQL Server machine, we don’t see the problem.
Looking into this error code:

err 0xc000006e 
# for hex 0xc000006e / decimal -1073741714 
STATUS_ACCOUNT_RESTRICTION ntstatus.h 
# Indicates a referenced user name and authentication 
# information are valid, but some user account restriction 
# has prevented successful authentication (such as 
# time-of-day restrictions). 

Here, 0xc000006e means STATUS_ACCOUNT_RESTRICTION, which indicates referenced user name and authentication information are valid, but some user account restriction has prevented successful authentication. If we add that user account to the local Administrators group, we are able to connect to SQL Server. We believe it is because we have “Audit: Shutdown system immediately if unable to log security audits” policy enabled. If HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Lsa\CrashOnAuditFail is 1, the local policy “Audit: Shut down system immediately if unable to log security audits” is enabled. When the security logs become full, the CrashOnAuditFail value will be changed to 2. At this point, only local administrators can logon and non-administrator accounts will fail.

We checked HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Lsa\CrashOnAuditFail and the value was 2
This problem occurs if the security event log has reached the maximum log size and the Event Log Wrapping setting is set to Overwrite Events Older than X Days or Do Not Overwrite Events. Because the security event log is full, and the CrashOnAuditFail registry key is set, Microsoft Windows does not permit accounts that are not administrator accounts to log on.

If the machine is a Web Server you may also see websites with anonymous access are being affected. When anonymous access is configured, requests to the Web site try to authenticate by using the IUSR_computername and IWAM_computername accounts. These accounts are not administrator accounts.

For further details please refer the following KB 832981

Microsoft Windows [Version 6.1.7600]
Copyright (c) 2009 Microsoft Corporation.  All rights reserved.

C:\Users\dipanb>cd \Program Files\Microsoft SQL Server\100\Tools\Binn

C:\Program Files\Microsoft SQL Server\100\Tools\Binn>sqlcmd -S GODFATHER1 -U testuser -P testuser -l 5 -Q"select 1"
HResult 0x102, Level 16, State 1
TCP Provider: Timeout error [258].
Sqlcmd: Error: Microsoft SQL Server Native Client 10.0 : Unable to complete login process due to delay in login response.
Sqlcmd: Error: Microsoft SQL Server Native Client 10.0 : Login timeout expired.

C:\Program Files\Microsoft SQL Server\100\Tools\Binn>

 As we can see if I reduce the connect timeout to 5 sec, we get a connection timeout. From a good client machine running on W2K3 Server or WinXp, if I set the timeout to 1 sec, I am still able to make a successful connection and I get a row returned. So what is different in the Windows 2008 Server machine? To find out what is happening, I took a Network trace using Microsoft Network Monitor 3.3

We can see a delay of approx 5 seconds in the trace. This tells us why we were getting a timeout if we reduced the connect timeout to 5 secs because the delay happened before the pre-login handshake.

The other customer facing this same issue was running a Linked Server query.

In this example, the Linked Server was on a Windows 2008 Server running from a SQL Server 2008 Sp1. It was querying a SQL Server 2005 running on Windows Server 2003. When I run the query through the Linked Server, it was taking 55 secs to return 64 rows. If I run the query directly from the 2005 SQL Server, the result is instantaneous. To troubleshoot this I took a Network trace from the Primary Server where the Linked Server is located and also on the Destination Server, from where the Linked Server is pulling data.

We are seeing the same 5 sec delay here as well.

Resolution

This delay may happen when your NIC Card or a router is not compatible with a few features which I am describing below:

The Scalable Networking Pack (SNP) feature includes TCP Chimney Offloading, Receive Side Scaling (RSS) and Network Direct Memory Access (NetDMA). For a Windows 2008 Server, TCP Chimney is disabled but RSS and NetDMA are enabled by default. The other factor which may contribute to the delay is Receive Window Auto Tuning feature which was introduced since Windows Vista SP1.

It is recommended to follow up with your network adapter vendor to see if they have an updated driver that will address the problem and allow the use of the above features. As an immediate relief you may turn these features off.

1. Receive side scaling (netsh int tcp set global rss=disabled) This was causing 15 sec latency.

2. Autotuning level (netsh interface tcp set global autotuninglevel=disabled) This was causing 52 seconds latency.

You can run the following command to get the TCP IP global values
netsh int tcp show global

After you disable the above settings your tcp global values should look like this:

Note: You may not see this behavior on all Windows 2008 Server when we have these values turned on. These happens only if you have an old router or NIC card and they are not compatible with these settings turned on.

More Information

https://support.microsoft.com/kb/951037

https://support.microsoft.com/kb/947239