Share via

Azure Data Factory Dataflow Costs

Devlin, Gannon 1 Reputation point
2022-08-25T18:58:32.683+00:00

I'm a bit confused on pricing and would really appreciate any clarification.
How is the same pipeline costing me $1.30 when ran thru debug but $6 when ran on a trigger?
235021-screenshot-94.png
234968-screenshot-95.png
I understand there is the cluster startup time but each dataflow is taking more than 10x longer and running up significantly more vcore hours
Is there anything I can do to reduce these costs besides purchasing reservations?
I'm running them on an general purpose IR with 8 cores and 0 ttl
I'd like to run the pipeline 4x a day but $24/day for one pipeline seems like a lot

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,196 questions

1 answer

Sort by: Most helpful
Most helpful Newest Oldest
  1. AnnuKumari-MSFT 32,161 Reputation points Microsoft Employee
    2022-08-26T07:12:13.807+00:00

    Hi @Devlin, Gannon ,

    Welcome to Microsoft Q&A platform and thanks for posting your question.

    I understand your query is related to billing of the Dataflow execution.

    Debug executions from pipelines and data preview debugging will continue to use the debug settings which has a preset TTL of 60 minutes.

    • If you set a TTL, ADF will maintain a pool of VMs which can be utilized to spin-up each subsequent data flow activity against that same Azure IR. This reduces the amount of time needed to start-up the environment before your job is executed.
    • ADF will maintain that pool for the TTL time after the last data flow pipeline activity executes. Note that this will extend your billing period for a data flow to the extended time of your TTL. However, your data flow job execution time will decrease because of the re-use of the VMs from the compute pool. The compute resources are not provisioned until your first data flow activity is executed using that Azure IR.
    • For more specialized assistance on this, I recommend you to create a billing support ticket which is free of cost. The ticket enables you to work closely with the support engineers and get a quick resolution to your issue.

    235057-image.png

    You can also contact Azure Billing/subscription support, which is free, and it's the best choice for your scenario.

    235099-image.png

    If you have questions or need help on how to create a support request, here are the steps to create support case: How to create a support ticket for Azure billing and subscription issues and OR Call to Global Customer Service phone numbers.

    Hope this will help. Please let us know if any further queries.

    ------------------------------

    • Please don't forget to click on 130616-image.png or upvote 130671-image.png button whenever the information provided helps you.
      Original posters help the community find answers faster by identifying the correct answer. Here is how
    • Want a reminder to come back and check responses? Here is how to subscribe to a notification
    • If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators