What is the maximum throughput or upload limit on the amount of simultaneous data being transferred using an Azure IRT?

RNA 1 Reputation point
2021-11-21T13:52:42.96+00:00

Is there a maximum throughput or upload limit on the amount of simultaneous data being transferred using an Azure IRT as part of an Azure data factory before a pipeline and/or activity may timeout or fail?

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,220 questions
SQL Server Integration Services
SQL Server Integration Services
A Microsoft platform for building enterprise-level data integration and data transformations solutions.
2,642 questions
{count} votes

1 answer

Sort by: Most helpful
  1. MartinJaffer-MSFT 26,226 Reputation points
    2021-11-23T04:24:13.36+00:00

    Hello @RNA and welcome to Microsoft Q&A.

    There is no "Microsoft policy imposed limitation" on your throughput that I know of. No data cap like in a mobile phone plan. That said, there are other limitations which happen.

    You can run into hardware or software limitations. The Integration Runtime lets you specify the compute power made available for Data Flows, and I have seen people sometimes push too much and make it choke. The Copy Activity lets you specify the number of Data Integration Units (DIU) (compute power used for transfer), and this mostly affects amount of parallelism.

    More often limits are hit on either the source or sink ability to accept connections.

    There are limits on number of concurrently running pipelines, concurrent external activities, and concurrent DIU. See azure-subscription-service-limits

    Correction: There is a maximum number of DIU in a single copy activity, and given that DIU affect the amount you can do in parallel, this is an indirect limit on simultaneous data.

    Data Integration Units per copy activity run 256

    1 person found this answer helpful.
    0 comments No comments

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.