Data Factory Copy task getting stuck on a specific record count when copying

Question

Hi All,

I am copying data from a SAP Hana table on Prem to parquet file in ADLS, the table is only 30 000 000 big but when the copy task get to 12 000 000 odd records it seems to hang or stops copying. I have checked the source data and it is fine no issues with it at all. I have changed all the setting possible within the copy task and have tried various ways of pulling the data.

My config on the copy task is as per the below:

Thank you in advance for assistance.

Answer 1

Hi @Stephen Smith

Thanks for the question and using MS Q&A platform.

Based on your description, it seems like the copy task is getting stuck at a specific record count when copying data from a SAP Hana table on Prem to a parquet file in ADLS. Here are some steps you can take to troubleshoot the issue:

1. Check the logs: Check the logs for the copy task to see if there are any errors or warnings that might indicate the cause of the issue.
2. Increase the timeout: Try increasing the timeout for the copy task to see if that resolves the issue. You can do this by setting the "Timeout" property in the copy task configuration.
3. Check the network: Check the network connectivity between the source and destination to ensure that there are no issues with the network.
4. Check the destination: Check the destination ADLS to ensure that there is enough space available for the data being copied.
5. Check the source data: Check the source data to ensure that there are no issues with the data being copied.
6. Try a different approach: Try a different approach to copying the data, such as using a different tool or method.

I hope this helps! Let me know if you have any further questions.