Errors When Copying Large Datasets from Dynamics SQL in Azure Data Factory

Question

What should I do if I encounter errors while copying data from a Dynamics SQL database using Azure Data Factory?

PS - Based on common issues that we have seen from customers and other sources, we are posting these questions to help the Azure community.

Answer 1

Greetings!
If you face errors such as "Cannot execute the required database operation" while copying data from a Dynamics SQL database using Azure Data Factory, the issue may be related to the large volume of data being processed. Initial findings indicated that more than 10 million rows were being copied, leading to long execution times and inconsistent results (some runs succeeded, others failed).

To resolve this, follow these steps:

1. Examine and Optimize Pipeline Settings: Ensure your pipeline settings are optimized for large data transfers.
2. Check SQL Server Logs: Investigate SQL server logs for any errors or issues that could be affecting the data copy operation.
3. Monitor Activity Runs: Use Azure Data Factory's logging and monitoring tools to track the status of activity runs and identify any patterns in failures.
4. Batch Data Transfer: If handling large volumes of data in a single operation is problematic, consider transferring the data in smaller batches to avoid timeouts and other issues.

For more detailed guidance, refer to the following Azure documentation:

• Copy and transform data in Azure Data Factory
• Troubleshoot the Copy Activity in Azure Data Factory

Hope this helps. If you have any follow-up questions, please let me know. I would be happy to help.

Please do not forget to "up-vote" wherever the information provided helps you, as this can be beneficial to other community members.