getting internal server error in Azure data flow

Question

Getting the following error :

Job failed due to reason: at Sink 'sink1': Failed to execute dataflow with internal server error, please retry later. If issue persists, please contact Microsoft support for further assistance

Activity ID 4e10fcfd-9527-48d8-8e37-85abe24d78d5

Any hint on how to debug this?

Answer 1

Hi javier cohenar ,

Thankyou for using Microsoft Q&A platform and thanks for posting your question here.

As per my understanding, you are getting internal server error while using mapping dataflow in Azure datafactory pipeline. Please let me know if that is not the case.

Kindly try creating integration runtime with higher compute size and use memory optimized compute type while executing dataflow activity and see if it helps.

Specific scenarios that can cause internal server errors are shown as follows.

Scenario 1: Not choosing the appropriate compute size/type and other factors

Successful execution of data flows depends on many factors, including the compute size/type, numbers of source/sinks to process, the partition specification, transformations involved, sizes of datasets, the data skewness and so on.

For more guidance, see Integration Runtime performance.

Scenario 2: Using debug sessions with parallel activities

When triggering a run using the data flow debug session with constructs like ForEach in the pipeline, multiple parallel runs can be submitted to the same cluster. This situation can lead to cluster failure problems while running because of resource issues, such as being out of memory.

To submit a run with the appropriate integration runtime configuration defined in the pipeline activity after publishing the changes, select Trigger Now or Debug > Use Activity Runtime.

Scenario 3: Transient issues

Transient issues with microservices involved in the execution can cause the run to fail.

Configuring retries in the pipeline activity can resolve the problems caused by transient issues. For more guidance, see Activity Policy.

For more details, kindly refer to below resources:

Internal server errors | Troubleshoot mapping data flows in Azure Data Factory

https://learn.microsoft.com/en-us/answers/questions/945338/df-executor-outofmemoryerror-azure-data-factory

Hope it helps. Kindly accept the answer by clicking on Accept answer button. Thankyou

Answer 2

thanks it turns out that the error was the way the output filen was named in the sink operation in the data flow activity. changing that fixed the issue

Answer 3

Hi javier cohenar ,

Glad that you figured out the way to resolve your issue and thank you for posting your solution so that others experiencing the same thing can easily reference this! Since the Microsoft Q&A community has a policy that "The question author cannot accept their own answer. They can only accept answers by others ", I'll repost your solution in case you'd like to "Accept " the answer.

It seems the issue was related to the sink filename . Providing a valid name to the output file resolved the issue.

Kindly accept the answer by clicking on Accept answer button so that the solution reaches the community in case anyone is facing similar issue. Thankyou.