Azure Data Factory self-hosted integration runtime performance

Question

Hi everyone, 

I had a performance review session with a customer and came back with the following questions:

We analyzed parallelization of activities and the customer asked me, why when they execute data sources import looping a Copy activity in parallel with over 100 data sources, the number of activities executed at a time is between 10 and 20. The Concurrent jobs metric shows usually 10 jobs in average and maximum 20 jobs from 30 available.

• 

Data is copied from source systems into an Azure SQL Database connected to the IR using private endpoints.

• We checked CPU and memory metrics for both, the azure SQL Database and the Self-hosted Integration Runtime and we cannot see any performance bottleneck, rather underutilization.
• They had regular VMs before for the SHIR, they changed to burstable VMs Serie B8ms a couple of month ago to try to save money. The performance remains the same. They would like to know how the max number of Concurrent jobs is calculated based on the VM size.
• They use a SHIR also for cloud data sources, because when they setup the platform, private endpoints were not supported in the Azure integration runtime. Should I advise them to switch the IRs?
• I tried to measure network latency and I/O bottleneck in different places (SHIR, private endpoints and Azure SQL DB) but have not found something relevant. Maybe someone can guide me to conduct a better networking analysis. 

Any comment will be highly appreciated 

Answer 1

Hi Paul Hernandez ,

The default value of the concurrent jobs limit is set based on the machine size. The factors used to calculate this value depend on the amount of RAM and the number of CPU cores of the machine. So the more cores and the more memory, the higher the default limit of concurrent jobs.

• You can monitor the performance of the integration runtime and adjust the maximum concurrent jobs based on the workload.
• You can scale out by increasing the number of nodes. When you increase the number of nodes, the concurrent jobs limit is the sum of the concurrent job limit values of all the available nodes.
• For example, if one node lets you run a maximum of twelve concurrent jobs, then adding three more similar nodes lets you run a maximum of 48 concurrent jobs (that is, 4 x 12). We recommend that you increase the concurrent jobs limit only when you see low resource usage with the default values on each node. You can override the calculated default value in the Azure portal. Select Author > Connections > Integration Runtimes > Edit > Nodes > Modify concurrent job value per node.
• The number of available concurrent jobs is determined by the size of the Self-hosted Integration Runtime (SHIR) VM. The B8ms VM size has 8 vCPUs and 32 GB of memory, which should be sufficient for running up to 30 concurrent jobs. However, burstable VMs have a CPU credit system that can limit their performance when the CPU usage exceeds the baseline level. This could be a factor in the performance issue.
• Since private endpoint doesn't support Azure IR, it would be better option to continue with SHIR but increasing the nodes by enabling more VMs to distribute the loads would be a better option. Max nodes that can be created is 4. When not in use, VM can be stopped and resources can be deallocated.

Hope it helps. Thankyou