Data Factory Copy task getting stuck on a specific record count when copying

Stephen Smith 0 Reputation points
2024-05-18T08:07:27.7766667+00:00

Hi All,

I am copying data from a SAP Hana table on Prem to parquet file in ADLS, the table is only 30 000 000 big but when the copy task get to 12 000 000 odd records it seems to hang or stops copying. I have checked the source data and it is fine no issues with it at all. I have changed all the setting possible within the copy task and have tried various ways of pulling the data.

My config on the copy task is as per the below:
User's image

User's image

User's image

User's image

Thank you in advance for assistance.

SAP HANA on Azure Large Instances
SAP HANA on Azure Large Instances
Microsoft branding terminology for an Azure offer to run HANA instances on SAP HANA hardware deployed in Large Instance stamps in different Azure regions.
120 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,899 questions
{count} votes

1 answer

Sort by: Most helpful
  1. Smaran Thoomu 11,370 Reputation points Microsoft Vendor
    2024-05-20T06:36:22.7233333+00:00

    Hi @Stephen Smith

    Thanks for the question and using MS Q&A platform.

    Based on your description, it seems like the copy task is getting stuck at a specific record count when copying data from a SAP Hana table on Prem to a parquet file in ADLS. Here are some steps you can take to troubleshoot the issue:

    1. Check the logs: Check the logs for the copy task to see if there are any errors or warnings that might indicate the cause of the issue.
    2. Increase the timeout: Try increasing the timeout for the copy task to see if that resolves the issue. You can do this by setting the "Timeout" property in the copy task configuration.
    3. Check the network: Check the network connectivity between the source and destination to ensure that there are no issues with the network.
    4. Check the destination: Check the destination ADLS to ensure that there is enough space available for the data being copied.
    5. Check the source data: Check the source data to ensure that there are no issues with the data being copied.
    6. Try a different approach: Try a different approach to copying the data, such as using a different tool or method.

    I hope this helps! Let me know if you have any further questions.

    0 comments No comments