ADF - Is there a minimum Manage VNET charge of one hour even if you used it a few minutes within the a time frame?

Question

In ADF is there ever a minimum charge for a whole hour? (rounded up)

In the past I've heard of ADF rounding up to the minute if activities take a few seconds. But I've never heard that ADF will round up our usage to an entire hour. I have been searching for documentation to this effect and cannot find any. However the ADF bills we are getting in Azure seems to imply that we are being charged for 24 hours of usage every day, even though the duration of our pipelines are lasting for a small fraction of the day. (we use ADF for just a few minutes each hour).

There is an adf "pricing concepts" page here:
https://learn.microsoft.com/en-us/azure/data-factory/pricing-concepts

Here is a quote:

Pipeline Activity = $0.116 (Prorated for 7 minutes of execution time. $1/hour on Azure Integration Runtime)

Based on this, it seems to imply that the hourly charge of $1 will be prorated and would cost a total of only ~12 cents in this scenario. This seems to indicate that we would NOT be charged for a whole hour, given a few minutes of usage.

Has anyone else seen their usage being rounded up, so that they are charged for an entire hour, rather than prorated based on their actual usage of ADF within that hour?

Answer 1

Yes, a customer's ADF bill may reflect a minimum managed VNET of an hour. This is based on the TTL of 60 mins for the "compute node" that hosts the VNET-enabled Azure IR.

The ADF "pricing concepts" page has now been updated to include the cost of this TTL. See:

https://learn.microsoft.com/en-us/azure/data-factory/pricing-concepts#data-integration-in-azure-data-factory-managed-vnet

This cost is especially significant when pipelines run at hourly intervals (despite the fact that they many only take a couple of minutes to complete.).

Answer 2

I was finally pointed to ADF documentation that describes a TTL related to managed virtual networks. This TTL will result in a minimum charge of one hour, regardless of whether your workload takes a minute or 60 minutes.

For non-copy activities including pipeline activity and external activity, there is a 60 minutes Time To Live (TTL) when you trigger them at the first time. Within TTL, the queue time is shorter because the node is already warmed up.

... at https://learn.microsoft.com/en-us/azure/data-factory/managed-virtual-network-private-endpoint#activity-execution-time-using-managed-virtual-network

Ideally this information would also be presented in the billing examples (pricing concepts). For our purposes the TTL seems extremely long and it greatly impacts the amount that we are charged for ADF. Here is another place where I think the information should be provided:

** ADF "pricing concepts" page :**
https://learn.microsoft.com/en-us/azure/data-factory/pricing-concepts#data-integration-in-azure-data-factory-managed-vnet

One issue that bothers me is that they claim there is a "compute node" that is warmed up for our VNET activities... but the ADF interface doesn't provide any visibility to confirm any of this. IE. If I'm being charged for some mysterious "compute node" at any given moment of time, then the ADF portal should also provide some visibility so we can inspect that compute node and see if it is actually running or not. The only visibility we are given is in our ADF billing at the end of the month. The ADF portal shows us activities, but doesn't allow us to see when our TTL is expiring (or whether it is expiring). Nor can we re-configure the default TTL of 60 minutes, based on what I'm hearing.

Based on this experience, it is clear that ADF VNET activities are going to be very expensive if you need to run one small pipeline on an hourly basis. Perhaps that is a factor that may lead customers to another Azure service, like appservices or azure functions. It isn't something that anyone warns you about, until you get the surprisingly high bill for the first time.