Scale up / scale down a self-hosted IR

Question

Scale up / scale down a self-hosted IR

Ravikiran Srini 20

In the para in the below link, it is mentioned that:

"When the processor and available RAM aren't well utilized, but the execution of concurrent jobs reaches a node's limits, scale up by increasing the number of concurrent jobs that a node can run. You might also want to scale up when activities time out because the self-hosted IR is overloaded"

https://learn.microsoft.com/en-us/azure/data-factory/create-self-hosted-integration-runtime?tabs=data-factory#scale-up

I do not understand the second line. Why would we increase the concurrency count (scale up) if the IR is overloaded? Does it mean to say scale down?

Ravikiran Srini 20 Reputation points

2024-10-06T01:07:57.9166667+00:00

Ok, I understand that in the second scale-out, they mean to add more RAM and CPU. There are two different meanings of scale-out implied in a single para, which is confusing.

Please let me know if my understanding is correct.
Ganesh Gurram 7,295 Reputation points Microsoft External Staff Moderator

2024-10-08T06:00:17.3033333+00:00

@Ravikiran Srini - We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution, please do share that same with the community as it can be helpful to others. Do let us know if you have any further queries.

1 answer

Your answer

Ravikiran Srini 20 Reputation points

2024-10-06T01:07:57.9166667+00:00

Ok, I understand that in the second scale-out, they mean to add more RAM and CPU. There are two different meanings of scale-out implied in a single para, which is confusing.

Please let me know if my understanding is correct.
Ganesh Gurram 7,295 Reputation points Microsoft External Staff Moderator

2024-10-08T06:00:17.3033333+00:00

@Ravikiran Srini - We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution, please do share that same with the community as it can be helpful to others. Do let us know if you have any further queries.

Answer 1

@Ravikiran Srini - Thanks for the question and using MS Q&A platform.

According to the documentation, when the self-hosted integration runtime (IR) is overloaded, we might want to scale up by increasing the number of concurrent jobs that a node can run. This means that if the processor and available RAM are not fully utilized, but the execution of concurrent jobs is reaching the limit, we can increase the number of concurrent jobs that can run on a node. This will help to optimize the performance of the self-hosted IR and prevent activities from timing out.

User's image Yes, you are correct. The documentation is using the term "scale out" in two different contexts, which can be interpreted differently by readers.

In the first context, "scale out" means adding more nodes (machines) to the self-hosted integration runtime to help distribute the load across multiple nodes. This is typically done when the processor usage is high and available memory is low on the self-hosted IR.

In the second context, "scale out" means increasing the number of concurrent jobs that a node can run. This is typically done when the processor and available RAM are not fully utilized, but the execution of concurrent jobs is reaching the limit.

Hope this helps. Do let us know if you have any further queries.

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.

Ganesh Gurram 7,295 Reputation points Microsoft External Staff Moderator

2024-10-09T05:23:41.19+00:00

@Ravikiran Srini - Just checking in to see if the above answer helped. If this answers your query, do click "Accept the answer” for the same, which might be beneficial to other community members reading this thread. And, if you have any further query do let us know.

Share via

Scale up / scale down a self-hosted IR

1 answer

Your answer