dataflow taking toomuch time for executions need help on priority.

Karnati,Venkata Suchendra Reddy,IN-Bangalore 6

Hi In my production pipeline i am using dataflow activity to falttern json file into csv . it taking too much time that is 45 minutes and still either fail or success.

same pipeline in dev environment same input data taking 4 to 5 min to complete. can anyone solve to fix this issues.

![172527-image.png][1]

![172561-image.png][3]

i have shared dataflow settings and dataflow skeyliton for your are understanding.

[1]: /api/attachments/172527-image.png?platform=QnA [3]: /api/attachments/172561-image.png?platform=QnA

Karnati,Venkata Suchendra Reddy,IN-Bangalore 6 Reputation points

2022-02-09T09:16:54.957+00:00

Here my source is data lake and sink is data lake.

after flattern 2 users and groups ,there is join and sink where time is not showing
AnnuKumari-MSFT 31,146 Reputation points Microsoft Employee

2022-02-09T10:20:05.877+00:00

Hi @Karnati,Venkata Suchendra Reddy,IN-Bangalore ,
Thankyou for using Microsoft Q&A platform and posting your query. Could you please check if there are any other pipelines running at the same time which might be sharing Integration Runtime that might cause delay. Also, Please compare your dev and Production pipeline JSON using online comparator tools . If it all looks good but issue still persists, then I would suggest you to raise a support ticket with Microsoft to analyze further on the issue. To get more details on how to raise the support ticket, kindly check the below article: Create an Azure support request
AnnuKumari-MSFT 31,146 Reputation points Microsoft Employee

2022-02-15T10:45:54.137+00:00

Hi @Karnati,Venkata Suchendra Reddy,IN-Bangalore ,
Following up to see if any of the below suggestions was helpful, kindly do click Accept Answer and Up-Vote for the same. In case you have any further query please do let us know.
AnnuKumari-MSFT 31,146 Reputation points Microsoft Employee

2022-02-16T06:47:50.037+00:00

@Karnati,Venkata Suchendra Reddy,IN-Bangalore ,
Just Following up to see if you got the resolution for the query. If not, please revert back so that we will respond back with the more details and we will try to help . If this answers your query, do click Accept Answer and Up-Vote for the same. And, if you have any further query do let us know.

2 answers

AnnuKumari-MSFT 31,146 Reputation points Microsoft Employee

2022-02-09T10:21:11.92+00:00

Hi @Karnati,Venkata Suchendra Reddy,IN-Bangalore ,

Thankyou for using Microsoft Q&A platform and posting your query. Could you please check if there are any other pipelines running at the same time which might be sharing Integration Runtime that might cause delay. Also, Please compare your dev and Production pipeline JSON using online comparator tools . If it all looks good but issue still persists, then I would suggest you to raise a support ticket with Microsoft to analyze further on the issue. To get more details on how to raise the support ticket, kindly check the below article: Create an Azure support request.

If this answers your query, do click Accept Answer and Up-Vote for the same. And, if you have any further query do let us know.
Please sign in to rate this answer.

0 comments No comments
Sign in to comment
MarkKromer-MSFT 5,186 Reputation points Microsoft Employee

2022-02-09T17:29:02.213+00:00

Look at the top portion of your activity monitoring view, which is cut-off and not showing in your screenshot here. It will tell you how much time it took to acquire the Spark cluster to execute your data flow. If it is in the 3-4 minute range, then you should set a TTL on your Azure IR so that the pipeline can use a warm cluster instead of requiring ADF to spin-up a new cluster on every data flow activity invocation.
Please sign in to rate this answer.
Karnati,Venkata Suchendra Reddy,IN-Bangalore 6 Reputation points

2022-02-10T04:50:42.527+00:00

could you please help us to to Set TTL.. generally auto ir is only option to choose. can u please guide in this scenario..

MarkKromer-MSFT 5,186 Reputation points Microsoft Employee

2022-02-10T05:10:45.887+00:00

Set a TTL that will keep the cluster around for the minimum amount of time between data flow activity executions. For example, if you run your data flows in sequence in a pipeline, set a small 10 minute TTL. If you execute data flows every hour, set a 60 minute TTL.

Karnati,Venkata Suchendra Reddy,IN-Bangalore 6 Reputation points

2022-02-10T05:47:34.77+00:00

even i tried with that still issue snot resolved even after 30 min still issues is there joining is not getting succeed .

Karnati,Venkata Suchendra Reddy,IN-Bangalore 6 Reputation points

2022-02-10T05:49:16.187+00:00

hi @MarkKromer-MSFT

same is working in dev environment but getting fail in prod . i were copied json code as it is ..

AnnuKumari-MSFT 31,146 Reputation points Microsoft Employee

2022-02-14T14:50:24.147+00:00

@Karnati,Venkata Suchendra Reddy,IN-Bangalore ,
Just checking back to see if you got the resolution for the query. If not, please revert back so that we will respond back with the more details and we will try to help . If this answers your query, do click Accept Answer and Up-Vote for the same. And, if you have any further query do let us know.
Sign in to comment