Azure ADF Pipeline Web Call Costs

Question

Hello We have a Azure Synapse setup with multiple pipelines that import data into the data factory.

We have recently added a pipeline that makes around 7000 calls a day to an Oracle API with each data packet around 500bytes.

Now for some reason we are getting billed 300 dollars a day for "cloud data movement" just on that pipeline alone.

Any suggestions on how to reduce this amount?

Answer 1

Hello @Jed Fletcher and welcome to Microsoft Q&A. $300 / day does sound quite concerning.

Lets try to break down the costs and see where things might be going wrong. I have a hunch that making many small calls (7000 x 500bytes) is not the best use of Data Factory, there might be a better tool for the job.

So 7000 x 500 bytes = 3500000 bytes = 3.5 megabytes. That is a very small amount. A single copy activity could do that for less than 1$ I think (very loose approximations here). So the total data isn't the issue.

There are 3 main modalities in which Data Factory charges you. Per-run and duration x compute. Duration x compute power is how the copy activity charges you. Per-run is for every time something happens, this can be a pipeline run, or an activity within a pipeline, or other stuff. Activity runs are $0.001 each. Multiply that by 7000 , thats 7$. So there must be more going on in your pipeline than that single call.

There is also data egress bandwidth costs, but that isn't specific to Data Factory or Synapse. That is an Azure wide thing.