Cost factors for Azure Data Factory

Question

Hi All,

I need to determine some factors w.r.t Azure Data Factory billing which is not available in any of Microsoft links.

1. Does inactive pipeline in ADF v2 have some default cost similar to how we had for ADF v1?
2. If Copy operation is configured in Upsert mode, when there is no change in the row, would Microsoft treat it as one modified entity and consider it for billing count?

Also, is there any way wherein we can create a spark cluster and reuse it in multiple Azure Data Factories for DataFlows rather than every ADF creating its own cluster for every data flow execution.

Note -- We have multiple ADF's around 20 to 25 wherein we want to reuse a cluster similar to azure DataBricks.

@ShaikMaheer-MSFT @AnnuKumari-MSFT @MartinJaffer-MSFT @PRADEEPCHEEKATLA-MSFT @KranthiPakala-MSFT

Answer 1

hey,
Based on my understanding:

1. ADF v2 charges for only active executions and not inactive ones like ADF v1
2. Since ADF v2 is actually moving a data and comparing a value in case of upsert , a row operation is actually being performed so ideally it would be considered chargeable
3. Unfortunately in dataflows , it is not possible to share clusters between diff ADFs

I may be wrong here and would let MSFT employees provide a final confirmation

Answer 2

Hello @Priya Jha ,

Thanks for the question and using MS Q&A platform.

Yes, agreed with @Nandan Hegde points. Thank you @Nandan Hegde for sharing your insights.

1. Does inactive pipeline in ADF v2 have some default cost similar to how we had for ADF v1? - No, In ADF v2, you only pay for data pipeline orchestration by activity run and activity execution by integration runtime hours (For IR charges are prorated by the minute and rounded up)
2. If Copy operation is configured in Upsert mode, when there is no change in the row, would Microsoft treat it as one modified entity and consider it for billing count? - Yes, since copy activity is being executed, that means you are performing Orchestration operations which incurs cost. Hence it is charged even though there are no valid rows to upsert.
3. Is there any way wherein we can create a spark cluster and reuse it in multiple Azure Data Factories for DataFlows rather than every ADF creating its own cluster for every data flow execution. - No, it is not possible in ADF. But if you have any specific feedback and requirements, I encourage you to please log your feedback in IDEAS forum here: https://feedback.azure.com/d365community/forum/1219ec2d-6c26-ec11-b6e6-000d3a4f032c and do share the link here so that other users with similar feedback can up-vote and comment on it to help increase the priority of the feature request. All the feedback shared in this forum are actively monitored and reviewed by respective product owners.

Hope this clarifies. If the above response from @Nandan Hegde answers your question, please do consider to accept it as helpful and up-vote which would help community to find the helpful answers.

Thank you

------------------------------

• Please don't forget to click on or upvote button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how