Timeout issue when using Dynamic range partition with big table

Question

Timeout issue when using Dynamic range partition with big table

Zhu, Yueli YZ [NC] 280

Hi I chose Dynamic range since the oracle source table is very large. And also set the degree of copy parallellism to 10. But I encounter a query timeout issue. I haven't choose the partition column since the primary key column is not sequential and update column contains null. So I haven't choose any partition lower and upper bound. Do you have any suggestions on the timeout issue and how to better configure the Dynamic range partition? User's image

User's image

AnnuKumari-MSFT 34,556 Reputation points Microsoft Employee Moderator

2024-01-22T05:48:01.6866667+00:00

Hi Zhu, Yueli YZ [NC] , Just following up to see if the below answer helped. Please do consider clicking Accept Answer as accepted answers help community as well. Also, please click on Yes for the survey 'Was the answer helpful'
AnnuKumari-MSFT 34,556 Reputation points Microsoft Employee Moderator

2024-02-01T10:23:06.0833333+00:00

Hi Zhu, Yueli YZ [NC] , Just checking if the below response helped you resolve your query, please do click Accept Answer

Accepted answer

0 additional answers

Your answer

AnnuKumari-MSFT 34,556 Reputation points Microsoft Employee Moderator

2024-01-22T05:48:01.6866667+00:00

Hi Zhu, Yueli YZ [NC] , Just following up to see if the below answer helped. Please do consider clicking Accept Answer as accepted answers help community as well. Also, please click on Yes for the survey 'Was the answer helpful'
AnnuKumari-MSFT 34,556 Reputation points Microsoft Employee Moderator

2024-02-01T10:23:06.0833333+00:00

Hi Zhu, Yueli YZ [NC] , Just checking if the below response helped you resolve your query, please do click Accept Answer

Answer 1

AnnuKumari-MSFT 34,556 Microsoft Employee Moderator

Hi Zhu, Yueli YZ [NC] , Thankyou for using Microsoft Q&A platform .

I understand that you are facing timeout issue while copying data from oracle source table and using dynamic partition.

Could you please share what is the volume of data in the source?

Additionally, kindly check the activity execution details to see in data transfer is in queued state entire time or it's in progress before the timeout. Kindly share the screenshot of the copy activity execution details.

Here are a few things you can try to resolve the timeout issue:

Increase the query timeout value: You can increase the query timeout value in the source dataset to allow more time for the query to complete. Go to the source dataset in ADF and set the "Query timeout" property to a higher value.
Reduce the degree of copy parallelism: If increasing the query timeout value does not resolve the issue, you can try reducing the degree of copy parallelism. This will reduce the number of concurrent queries that are being executed and may help to avoid query timeouts.

Hope it helps. Kindly share the required details and accept the answer if it helped. Thankyou

Zhu, Yueli YZ [NC] 280 Reputation points

2024-01-16T17:02:41.1433333+00:00

The column of data is 178 GB. And it was in progress before timeout.
Zhu, Yueli YZ [NC] 280 Reputation points

2024-01-16T21:36:36.5333333+00:00

Also, I increased the query timeout value and reduce the degree of copy parallelism to 8, it got Oracle unexpected network error
AnnuKumari-MSFT 34,556 Reputation points Microsoft Employee Moderator

2024-01-18T06:03:53.5033333+00:00

Hi Zhu, Yueli YZ [NC] , Thanks for the updates after trying the above suggested approach.
I understand that you are facing "Unexpected network error" in oracle side. Could you please work with your database administrator and try changing the data integrity settings of your Oracle server and see if that helps: Very intermittent copy activity fails with ORA-12656: Cryptographic checksum

Kindly check this thread where one of the community member was able to get the similar issue resolved by adding dataintegritylevel=0 in the connection string.
Hope it helps. Kindly accept the answer by clicking on Accept answer button. Thankyou
Zhu, Yueli YZ [NC] 280 Reputation points

2024-02-01T19:25:05.59+00:00

Thanks for your answers and help. With dynamic range for the big tables, still have issue. So, I work around to use batch load instead of dynamic range.
AnnuKumari-MSFT 34,556 Reputation points Microsoft Employee Moderator

2024-02-02T06:03:24.4033333+00:00

Zhu, Yueli YZ [NC] , Glad to know you were able to resolve the issue. Thanks for sharing the resolution with the community

Share via

Timeout issue when using Dynamic range partition with big table

0 additional answers

Your answer