adf completly unefficient

Question

i created an etl job with adf

something simple

• Extract : list of files from blobstorage format csv
• Transform : for each source file, create a new csv with 2 columns mapped including one simple calculated column
• Load : push new files into 2 target blobstorages

For one file of 215B (15 rows) to transform, it take 5 min.
how it is possible to be so unefficient? for something so simple, it should take at worst, 5 sec

ADF is supposed to be able proceed large amount of datas.
In next etl job i will need to proceed, i will have more than 400k rows...

Answer 1

Hey, ADF by default takes some time to create the dataflow cluster at run time. So that would be same irrespective of the number of rows in your file.
This might be the case why it took 5 mins.

similar thread:
https://stackoverflow.com/questions/63933355/azure-data-factory-transformation-pipeline-take-more-than-5-minutes-to-execute-w