How to fix the Pipeline which is running as expected in ADF dev, post which I have published the pipeline (and associated objects) to ADF Prod it fails.

Question

Hello Team,

 

I have build ADF pipeline to load data from source (SQL server) to ADLS.

Pipeline is running as expected in ADF dev, post which I have published the pipeline (and associated objects) to ADF Prod.

 

But while running the pipeline (Debug), the jobs runs for long time and fails doing nothing. (1hr 40mins)

 

Error msg:

Operation on target ForEach_Table failed: Activity failed because an inner activity failed; Inner activity name: Load data source to ADLS, Error: ErrorCode=SystemErrorActivityRunExecutedMoreThanOnce,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=The activity run failed due to service internal error, please retry this activity run later.,Source=Microsoft.DataTransfer.TransferTask,'

Thanks in Advance.

Regards,

Nikhil

Answer 1

Hi @Bhosle, Nikhil
Welcome to Microsoft Q&A platform and thanks for posting your question here.

It is possible that the pipeline is working in the development environment but not in the production environment due to differences in the configuration of the two environments. You should check the configuration settings of both environments and ensure that they are the same.

Regarding the long running time of the pipeline, it could be due to various reasons such as network latency, data volume, or resource constraints. You should monitor the pipeline run and optimize the pipeline by using techniques such as partitioning, compression, or caching.

The error message you have shared states that the ForEach_Table activity failed due to an inner activity failure. This error can occur due to various reasons such as network issues, credential failures, or service outages. You should check the logs and monitor the pipeline run to identify the root cause of the issue.

To resolve this issue, you can try the following steps:

• Check the configuration settings of both environments and ensure that they are the same.
• Monitor the pipeline run and optimize the pipeline by using techniques such as partitioning, compression, or caching.
• Check the logs and monitor the pipeline run to identify the root cause of the issue.
• Retry running the pipeline in production multiple times to check if the issue is a one-time occurrence. ADF has a retry mechanism that can potentially fix the problem on subsequent attempts.
• Ensure that the Integration Runtime is running and connected to the correct network with the necessary permissions and credentials to access the required resources.

I hope this information helps you. Let me know if you have further questions.

Answer 2

Hello Team,

I checked the conf for both dev and prd and it same. Is their anything else i need to check?

Regards,

Nikhil

Answer 3

I have created a new IR in dev with production settings

Also encountered this error while debugging:

Answer 4

Hello @Bhosle, Nikhil

This error message suggests that the activity run failed due to a service internal error, and you should retry this activity run later.

Here are some possible reasons for this error:

• Transient network issues
• Credential failures
• Service outages

To resolve this issue, you can try the following steps:

1. Rerun the pipeline and activities as described in the Azure Data Factory documentation.
2. Check if there are any transient network issues, credential failures, or service outages.
3. If the issue persists, create a support case. If you need more information, please let me know.

---

I hope that this response has addressed your query and helped you overcome your challenges. If so, please mark this response as Answered. This will not only acknowledge our efforts, but also assist other community members who may be looking for similar solutions.