Network error code 0x80090330 - existing connection was forcibly closed by the remote host

Pds 46

Hi,
We have on going issue within our one of the job running every few minutes but every day it's keep failing 2-3 times randomly, no specific timing.
Basically Client m/c running BCP job which trying contact target database sql server 2014.
We checked TLS and it's all same on both Client and server.
We also have updated ODBC drive 13.
When I am checking the Sql Server Error log, every time it shows following error on Server:

Error:
Network error code 0x80090330 occurred while establishing a connection; the connection has been closed. This may have been caused by client or server login timeout expiration. Time spent during login: total 136 ms, enqueued 0 ms, network writes 0 ms, network reads 1 ms, establishing SSL 134 ms, network reads during SSL 98 ms, network writes during SSL 0 ms, secure calls during SSL 36 ms, enqueued during SSL 0 ms, negotiating SSPI 0 ms, network reads during SSPI 0 ms, network writes during SSPI 0 ms, secure calls during SSPI 0 ms, enqueued during SSPI 0 ms, validating login 0 ms, including user-defined login processing 0 ms. [CLIENT: 10.X.X.X)

When I check log where our job running, I see following error:

SQLState = 08001, NativeError = 10054
Error = [Microsoft][ODBC Driver 13 for SQL Server]SSL Provider: An existing connection was forcibly closed by the remote host.
SQLState = 08001, NativeError = 10054
Error = [Microsoft][ODBC Driver 13 for SQL Server]Client unable to establish connection

Any idea what happening when job is failing?

Erland Sommarskog 100.9K Reputation points MVP

2021-03-13T10:31:25.53+00:00

Using the old Error Lookup utility, I see that error 0x80090330 supposedly means "The specified data could not be decrypted. "

I will need to confess that I am not sure what this implies. But generally, spurious errors like this one, are often a token to a problem in the network structure. Like a switch flipping a few bits or somesuch.
Pds 46 Reputation points

2021-03-14T14:47:35.08+00:00

Thank you Erland.
So what exactly you are suggesting and how?
Again, Network failure error is random.
Erland Sommarskog 100.9K Reputation points MVP

2021-03-14T20:18:45.813+00:00

I'm not suggesting much. I only wanted to provide the likely meaning of the error code.

You say that this is a job. But that is not an agent job? Does this only happen with BCP? Do any other connections to SQL Server run from this machine?
Pds 46 Reputation points

2021-03-16T14:06:17.833+00:00

Thanks Erland.
Mostly with the BCP job but we have couple T-Sql block that also failing but like BCP failure.
Pds 46 Reputation points

2021-03-16T14:07:33.227+00:00

Also it;s not a Agent job.
Erland Sommarskog 100.9K Reputation points MVP

2021-03-16T20:50:17.42+00:00

Mostly with the BCP job but we have couple T-Sql block that also failing but like BCP failure.

As I said, this sounds like a spurious problem in your network infrastructure. Can surely be difficult to track down.
Pds 46 Reputation points

2021-03-16T21:02:10.46+00:00

Thanks Erland.
I agreed that it will be difficult that's why trying get expert advice.
Tried different way troubleshooting but couldn't find right solution.

6 answers

crlpr-mcsa-1314 1 Reputation point

2021-03-13T04:02:47.413+00:00

Hi!
Try to disable TTL for all connections entrance to the database.
Hope it help.

If this answer ok, mark as resolved and Upvoted
Please sign in to rate this answer.
Pds 46 Reputation points

2021-03-14T14:57:32.19+00:00

Thank you!
So you mean to disable TTL in source server?
I was looking for TTL and couldn't find how to do it, it was also showing something for Always ON and AG listener, this is not a configured for AG, it's just both Regular Sql server and running BCP utility.

pdsqsql 391 Reputation points

2021-04-15T20:10:49.167+00:00

Could you please suggest how to disable TTL for all connections entrance to the database?
Sign in to comment
AmeliaGu-MSFT 13,961 Reputation points Microsoft Vendor

2021-03-15T08:27:22.5+00:00

Hi @Pds ,

What the version of SQL Server and OS are you using?
Could you please try to install SP3 and SP3 CU4 for SQL Server 2014 if you have not installed them?
Please refer to TLS 1.2 support for Microsoft SQL Server which might help.

Best Regards,
Amelia
Please sign in to rate this answer.
Pds 46 Reputation points

2021-03-16T14:04:11.127+00:00

Thanks Amelia.
For Target Server - Sql Server 2014 - SP3-CU-GDR (12.0.6372.1) and Windows Server 2016 Standard Version 1607

pdsqsql 391 Reputation points

2021-04-15T16:42:13.023+00:00

Thanks Amelia.
We have same SSL and TLS configuration on both Client and Server.
We are getting every day error as job is running frequently every day and it's failing few times so not able to exact issue.

In Sql error log, we have following error:

Blockquote
Network error code 0x80090330 occurred while establishing a connection; the connection has been closed. This may have been caused by client or server login timeout expiration. Time spent during login: total 111 ms, enqueued 0 ms, network writes 0 ms, network reads 0 ms, establishing SSL 111 ms, network reads during SSL 73 ms, network writes during SSL 0 ms, secure calls during SSL 37 ms, enqueued during SSL 0 ms, negotiating SSPI 0 ms, network reads during SSPI 0 ms, network writes during SSPI 0 ms, secure calls during SSPI 0 ms, enqueued during SSPI 0 ms, validating login 0 ms, including user-defined login processing 0 ms. [CLIENT:XX.X.X.XXX]

pdsqsql 391 Reputation points

2021-04-15T17:00:26.45+00:00

Thanks Amelia.
We have same SSL and TLS configuration on both Client and Server.
We are getting every day error as job is running frequently every day and it's failing few times so not able to exact issue.

In Sql error log, we have following error:

Blockquote
Network error code 0x80090330 occurred while establishing a connection; the connection has been closed. This may have been caused by client or server login timeout expiration. Time spent during login: total 111 ms, enqueued 0 ms, network writes 0 ms, network reads 0 ms, establishing SSL 111 ms, network reads during SSL 73 ms, network writes during SSL 0 ms, secure calls during SSL 37 ms, enqueued during SSL 0 ms, negotiating SSPI 0 ms, network reads during SSPI 0 ms, network writes during SSPI 0 ms, secure calls during SSPI 0 ms, enqueued during SSPI 0 ms, validating login 0 ms, including user-defined login processing 0 ms. [CLIENT:XX.X.X.XXX]
Sign in to comment
Erland Sommarskog 100.9K Reputation points MVP

2021-04-15T21:09:42.953+00:00

Not much help, but I looked up the code 0x80090330 the in the old Error Lookup utility and it tells me /The specified data could not be decrypted. /

But I don't think it is a TLS issue, because I would not expect such errors to be intermittent.

It could be something in your network that is eating bits and thereby disrupting communication. Although in that case, you could not only see errors on login. Unless the issue is in with the communication the AD?
Please sign in to rate this answer.

0 comments No comments
Sign in to comment
pdsqsql 391 Reputation points

2021-04-15T21:37:01.147+00:00

Thanks Erland.
I even don't think so any TLS issue.
It's strange that randomly having issue so couldn't find it out but every time it has Network Error code 0x80090330 because I turned on Trace Flag otherwise I am not gonna know the error code.

Any Driver related Issue?

I can see in my Job scheduler following Error whenever I see error code 0x80090330.

SQLState = 08001, NativeError = 10054 Error = [Microsoft][ODBC Driver 13 for SQL Server]SSL Provider: An existing connection was forcibly closed by the remote host.
SQLState = 08001, NativeError = 10054
Error = [Microsoft][ODBC Driver 13 for SQL Server]Client unable to establish connection
Please sign in to rate this answer.
Erland Sommarskog 100.9K Reputation points MVP

2021-04-16T21:50:43.307+00:00

Unlikely to be a driver issue, since it's intermittent. But if the driver has an allocation error, and has allocated less memory than it writes, the far end could be scribbled, And, well, if it was the network, errors could be all over the place, but a miss-allocation issue is likely to be in a single place, and therefore the failure can only happen at a certain stage.

You could try a later version of the ODBC driver to see if that helps.

The error you get in the agent job is likely to be due to what you see in the SQL Server errorlog. That is, after this error, SQL Server decides that it does not want to talk any further and closes the connection.

pdsqsql 391 Reputation points

2021-04-16T22:13:27.257+00:00

Thanks Erland.
If it's driver issue then it won't work at all, right?
Job is running all the time just failing 2-3 or 3-5 times out of 7000 times.

Driver which version? on which server - Job scheduler Server or Sql server?
We have Sql server 2014.

Erland Sommarskog 100.9K Reputation points MVP

2021-04-16T22:19:29.127+00:00

Yes, that was I said in my first sentence. But I then went to discuss the possibility of a scribbling error. That is, code that writes to memory which is not allocated, at least not to the buffer being written to. Such bugs can cause intermittent errors in software. So it can be worth trying a new version of the driver.
Sign in to comment
pdsqsql 391 Reputation points

2021-04-16T22:26:05.333+00:00

Thanks once again Erland.

So which version of driver you want me to update?
It will be on Sql server 2014, right or both Job scheduler server (where code getting executed) ?
Do you think that actual Server Memory also causing issue as we have total 8 GB and 6 GB is allocated?
Please sign in to rate this answer.
Erland Sommarskog 100.9K Reputation points MVP

2021-04-16T22:29:35.693+00:00

I don't want anything. It's not my server. :-)

As I said above, a later version. I think the most recent is 17.4, but I don't feeling like googling it - you could do that as well.

If my speculation is correct - please understand that it is a shot in the dark - the server memory does not matter. This would be a accident that occurs in the virtual address space of the driver.
Sign in to comment