Overwrite tables alone failing on a daily basis when triggered

Nanda Kumar Mohan 1

My pipeline is having many tables, out of which 2 tables are overwrite tables and either of the 1 table fails for sure on a daily basis stating insufficient information to connect to source error, but if i use rerun from failed state or if rerun the entire pipeline again it succeeds, not sure what is the problem, i tried retry option but that never worked in my scenario, can someone provide a solution for this, or tell me like how can automate rerun from failed state activity in ADF.

MartinJaffer-MSFT 26,021 Reputation points

2022-09-30T00:06:09.187+00:00

Hello and welcome to Microsoft Q&A @Nanda Kumar Mohan

Let me see if I understand right, your pipeline has 2 tables of a specific type, and every run, one of them fails. Which one of the two tables changes.
I haven't seen an error message stating insufficient information to connect to source error . Could you please share with me the full error message?
Also, what is your source and sink? Is this Copy activity or Dataflow or something else?
Nanda Kumar Mohan 1 Reputation point

2022-10-03T13:31:41.313+00:00

Failure happened on 'Source' side. ErrorCode=UserErrorFailedToConnectOdbcSource,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=ERROR [HY000] [Microsoft][ODBC Oracle Wire Protocol driver]Insufficient information to connect to the data source.\r\nERROR [HY000] [Microsoft][ODBC Oracle Wire Protocol driver]Insufficient information to connect to the data source.,Source=Microsoft.DataTransfer.ClientLibrary.Odbc.OdbcConnector,''Type=System.Data.Odbc.OdbcException,Message=ERROR [HY000] [Microsoft][ODBC Oracle Wire Protocol driver]Insufficient information to connect to the data source.\r\nERROR [HY000] [Microsoft][ODBC Oracle Wire Protocol driver]Insufficient information to connect to the data source.,Source=,'",
"failureType": "UserError",
"target": "IngestDBdata_to_ADLS",
"details": []
Nanda Kumar Mohan 1 Reputation point

2022-10-03T13:34:09.933+00:00

It's a copy activity and it fails during the copy activity only and if i do rerun from failed state using manual way it gets succeeded, this too 1 time it runs and next time it throws same error and fails again. Not able to find the root cause nor the solution as well, i tried triggering the pipeline at different time frames still it fails and every Friday alone it is getting passed other days for sure failing.
MartinJaffer-MSFT 26,021 Reputation points

2022-10-04T16:58:37.41+00:00

Hmm. by any chance is the Linked Service or dataset parameterized @Nanda Kumar Mohan ?

Also, did the copy activity take a long time?

According to non MSFT site, this can happen during a connection failover.

You are saying it works fine only on Fridays? Interesting. Might Fridays correlate with less load on your database / server? Working only on certain days of the week cuts away many possible causes, assuming everything else stays the same. Heavy workload or network issues might cause the network failover. One way to test, is to schedule the run for off-peak hours.
Nanda Kumar Mohan 1 Reputation point

2022-10-07T10:21:37.183+00:00

I tried on multiple timezones(peak hours too) one time it is getting passed on the other it is failing, i have used until activity to make it run till it gets passed but still the loop runs infinitely still it is not getting passed.
MartinJaffer-MSFT 26,021 Reputation points

2022-11-15T18:17:06.233+00:00

Let me know if you need a support ticket for this @Nanda Kumar Mohan