Share via

200+ columns assert in data flow is not executing even with 16 cores IR

saideepak obbineni 1 Reputation point
2022-02-22T15:02:22.267+00:00

I have a data flow which needs to validate 200+ columns in Azure Data Factory. My source file is Excel and I am using 'Assert' to validate the columns. When I gave 5 columns to validate in assert, I am able to preview data in debug mode. But when I used all the 200+ columns validations, it just shows 'Fetching Data' and then timeouts. Can someone please help how to achieve this quick. I have published the flow and tried to execute it on 16 cores Integration runtime. Still it is of no use. I terminated the flow after waiting for 30 minutes. The size of my data set is quite small around 20 rows.

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,689 questions

1 answer

Sort by: Most helpful
Most helpful Newest Oldest
  1. ShaikMaheer-MSFT 38,441 Reputation points Microsoft Employee
    2022-02-24T15:42:38.797+00:00

    Hi @saideepak obbineni ,

    Thank you for posting query in Microsoft Q&A Platform.

    As per my understanding, assert transformation with 200+ columns validation is getting timeout with 16 cores. Kindly correct me if it is wrong.

    Could you please try increasing core size even more and see if that helps?
    177611-image.png

    Please let us know how it goes.

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.