Decrease ADF self-hosted IR concurrent jobs

Question

I am reading about ADF self-hosted IR and microsoft documentation explains when to scale up.

While I understood that the default value is determined based on the machine size, I was unable to find an answer to the following question:

If we consistently have fewer running jobs than the concurrent job limit, would there be any advantage in decreasing the concurrent jobs limit? For example, would it help in releasing unused resources?

Thanks in advance for the support

Answer 1

Hi Lakshika ,

Welcome to Microsoft Q&A platform and thanks for posting your query here.

You can go ahead and decrease the limit for concurrent jobs, however, there won't be any significant difference .If you have fewer running jobs than the concurrent job limit, the integration runtime will not allocate any additional resources to the nodes, and the unused resources will still remain idle .

• You can monitor the performance of the integration runtime and adjust the maximum concurrent jobs based on the workload. The default value of the concurrent jobs limit is set based on the machine size. The factors used to calculate this value depend on the amount of RAM and the number of CPU cores of the machine. So the more cores and the more memory, the higher the default limit of concurrent jobs.
• You can scale out by increasing the number of nodes. When you increase the number of nodes, the concurrent jobs limit is the sum of the concurrent job limit values of all the available nodes.
• For example, if one node lets you run a maximum of twelve concurrent jobs, then adding three more similar nodes lets you run a maximum of 48 concurrent jobs (that is, 4 x 12). We recommend that you increase the concurrent jobs limit only when you see low resource usage with the default values on each node.

Additional resources:

Concurrent jobs limit

Troubleshooting guide for SHIR

Hope it helps. Kindly accept the answer if it's helpful . Thankyou