ADF Mapping Dataflows performance very low as compared to Databricks when performing same set of transformations.

amit singh 1

I am doing same set of transformation using ADF Mapping Dataflows and Databricks. Dataflows takes around 2 hours whereas Databricks is taking only 8-10 mins for the same volume of data. As i understand they both use Apache Spark in the back-end, can someone please help me understand this behavior.

HarithaMaddi-MSFT 10,136 Reputation points

2020-07-27T15:32:12.94+00:00

Hi @amit singh ,

Welcome to Microsoft Q&A Platform.

In order to investigate further, our internal team requested for below details, can you please send mail to AzCommunity [at] microsoft [.] com in the below format.

Subject of the email: <Attn - Haritha : ADF Mapping Dataflows performance very low as compared to Databricks when performing same set of transformations.>
Thread URL: https://learn.microsoft.com/en-us/answers/questions/52760/index.html
Support files for dataflow and run id: <>
Notebook if can be shared: <>
HarithaMaddi-MSFT 10,136 Reputation points

2020-07-29T05:22:56.877+00:00

Hi @amit singh ,

We have not received a response from you. Are you still facing the issue? If you found a solution, would you please share it here with the community? Otherwise, please share the details as stated above and we will continue to engage with you on the issue.
HarithaMaddi-MSFT 10,136 Reputation points

2020-07-30T12:50:42.06+00:00

Hi @amit singh ,

We have not received a response from you. Are you still facing the issue? If you found a solution, would you please share it here with the community? Otherwise, please share the details as stated above and we will continue to engage with you on the issue.
amit singh 1 Reputation point

2020-08-11T07:16:51.29+00:00

@HarithaMaddi-MSFT I have mailed the details at AzCommunity as asked.
HarithaMaddi-MSFT 10,136 Reputation points

2020-08-11T07:28:48.9+00:00

Thanks @amit singh , we have received the mail. I have forwarded the details to Internal team to investigate further. If possible please share pipeline run id from ADF where dataflow is taking ~2hours that will help to dig into issue quickly.
HarithaMaddi-MSFT 10,136 Reputation points

2020-08-19T06:17:26.37+00:00

Hi @amit singh ,

As our product engineer requested additional information from you in the mail, request you to share them for further investigation. If you found a solution, would you please share it here with the community?
HarithaMaddi-MSFT 10,136 Reputation points

2020-08-24T05:15:15.407+00:00

Hi @amit singh ,

We still have not heard back from you. We are waiting for inputs from you as requested by our product engineer in the mail for further investigation. If you found a solution, would you please share it here with the community?
Pranita Naik 1 Reputation point

2021-02-05T19:49:30.64+00:00

@amit singh , @HarithaMaddi-MSFT
Did we get the reason for this huge difference in execution in ADF vs that in Databricks?
Please share the analysis, would be helpful to decide the data strategy.

Share via

ADF Mapping Dataflows performance very low as compared to Databricks when performing same set of transformations.

Your answer