Azure Data Factory Performance

Question

Azure Data Factory Performance

Nilesh Omprakash Agrawal 10

Hello,

We have noticed that Azure Data Factory is extremely slow recently. A simple switch case that should take not more than few to resolve is not more that 20 minutes in some cases. I am attaching screenshot of one such run where you see that it is more than 24 minutes and yet trying to resolve switch case. This has been impacting our loading times and our customer has to wait long time to get refreshed data. Please note that we havent made any changes in logic since last week when things were running very smooth. We have tried to escalate this MS support but still havent received any constructive feedback. So, i am wondering if there are others too who has noticed slow performance of ADF this week?

User's image

AnnuKumari-MSFT 34,556 Reputation points Microsoft Employee Moderator

2023-01-20T04:59:10.1033333+00:00
Hi Nilesh Omprakash Agrawal ,

Welcome to Microsoft Q&A platform and thanks for posting your question here.

As I understand your issue, you are facing slowness in ADF pipeline execution since past few days that too without any explicit changes in the pipeline. Please let me know if that is not correct understanding.

For better understanding, could you please give more details around it. For instance:

What exactly is switch activity doing ? Is the dataset on which it is checking the conditions huge? Basically, how large volume data are you dealing with?

Which integration runtime is the pipeline dependent on? Is it Auto-resolve or self hosted?Probably it would have less impact as I can see activities inside switch are not being called upon. Switch condition itself is getting processed for more than 20 minutes.

It would be better if you share the switch activity expression screenshot for better understanding. Thankyou.
AnnuKumari-MSFT 34,556 Reputation points Microsoft Employee Moderator

2023-01-23T11:08:33.88+00:00

Hi Nilesh Omprakash Agrawal ,

Just checking to see if you got a chance to visit the previous response? Please revert back with the required details as asked above. Thankyou.
AnnuKumari-MSFT 34,556 Reputation points Microsoft Employee Moderator

2023-01-25T09:34:21.0633333+00:00

Hi Nilesh Omprakash Agrawal ,

We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet .

In case if you have any resolution please do share that same with the community as it can be helpful to others . Otherwise, please respond back with the more details and we will try to help
AmolRajmane-0617 1 Reputation point

2024-02-12T11:41:36.1066667+00:00

Data factory really performing bad with iteration and conditional activities. It takes lot of to come out of existing loop and start next iteration. I am facing this since many years and the problem is still exists.
Nilesh Omprakash Agrawal 10 Reputation points

2024-02-12T11:45:11.1566667+00:00

Hi, I have been noticing very bad performance with Switch case. I have an ongoing MS ticket for this issue. I hope to get some resolution from them.

1 answer

Your answer

AnnuKumari-MSFT 34,556 Reputation points Microsoft Employee Moderator

2023-01-20T04:59:10.1033333+00:00

Hi Nilesh Omprakash Agrawal ,

Welcome to Microsoft Q&A platform and thanks for posting your question here.

As I understand your issue, you are facing slowness in ADF pipeline execution since past few days that too without any explicit changes in the pipeline. Please let me know if that is not correct understanding.

For better understanding, could you please give more details around it. For instance:

What exactly is switch activity doing ? Is the dataset on which it is checking the conditions huge? Basically, how large volume data are you dealing with?

Which integration runtime is the pipeline dependent on? Is it Auto-resolve or self hosted?Probably it would have less impact as I can see activities inside switch are not being called upon. Switch condition itself is getting processed for more than 20 minutes.

It would be better if you share the switch activity expression screenshot for better understanding. Thankyou.
AnnuKumari-MSFT 34,556 Reputation points Microsoft Employee Moderator

2023-01-23T11:08:33.88+00:00

Hi Nilesh Omprakash Agrawal ,

Just checking to see if you got a chance to visit the previous response? Please revert back with the required details as asked above. Thankyou.
AnnuKumari-MSFT 34,556 Reputation points Microsoft Employee Moderator

2023-01-25T09:34:21.0633333+00:00

Hi Nilesh Omprakash Agrawal ,

We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet .

In case if you have any resolution please do share that same with the community as it can be helpful to others . Otherwise, please respond back with the more details and we will try to help
AmolRajmane-0617 1 Reputation point

2024-02-12T11:41:36.1066667+00:00

Data factory really performing bad with iteration and conditional activities. It takes lot of to come out of existing loop and start next iteration. I am facing this since many years and the problem is still exists.
Nilesh Omprakash Agrawal 10 Reputation points

2024-02-12T11:45:11.1566667+00:00

Hi, I have been noticing very bad performance with Switch case. I have an ongoing MS ticket for this issue. I hope to get some resolution from them.

Answer 1

McDermott_Ann 0

I dont have an answer, just another question. Why is Data Factory so slow and how can we get it to increase the performance? I don't see options for increasing your Data Factory server sizes or have any choices for building a bigger Factory. Please help. I have spent months on creating Pipelines and performance is so bad that its almost unusable.

AnnuKumari-MSFT 34,556 Reputation points Microsoft Employee Moderator

2023-08-19T09:51:13.2833333+00:00

McDermott_Ann ,

Recommending you to create a new post for better engagement. Thankyou

Share via

Azure Data Factory Performance

1 answer

Your answer