ADF D365 copy intermittent error

Question

Hi,

I use the Data Factory pipeline template for copying Dynamics data exported through Synapse Link. I've been getting an intermittent error when the Dataflow inside the template tries to run.

Internal Server Error in Synapse batch operation: 'Failed to send request to Livy due to exception=[System.Threading.Tasks.TaskCanceledException].'.

Are there any issues that have not been reported for either ADF or D365?

Thank you

Diana

Answer 1

There is failure in communication with the Synapse service, particularly with the Livy service, which is responsible for managing Spark jobs in Synapse.Sometimes, Azure services can experience throttling or become overloaded, especially if many requests are being processed simultaneously. This could cause delays or failures in the Livy service when submitting Spark jobs.

TaskCanceledException often points to operations exceeding their timeout period. You may want to check the timeout settings in your Data Factory pipeline and consider increasing them if they are too short.

If these steps do not resolve the issue, I recommend reaching out directly to Microsoft Support with detailed logs and descriptions of when the issue occurs.

Answer 2

@Diana Rus

Thanks for the question and using MS Q&A platform.

please make sure the below checks

Monitor Resource Utilization: Check if there are any resource constraints or spikes in usage that might be causing the issue. Sometimes, increasing the allocated resources can help.

Network Stability: Ensure that there are no network issues between your Data Factory and Synapse services. Network instability can sometimes cause communication failures.

Update Dependencies: Make sure all your services and dependencies are up to date. Sometimes, updates can resolve underlying issues.

Error Handling: Implement robust error handling in your pipeline to catch and log detailed error information. This can help in diagnosing the issue more effectively.

If the problem persists, please provide the detailed logs and error messages and screenshots if possible