Dynamic load balance

Question

Dynamic load balance

Ryan Abbey 1,186

We have a Synapse Data Factory process that dynamically kicks off >200 extracts. As Data Factory has a 20 pipeline running concurrently limit, the remaining extracts are queued behind the initial 20

However, it appears to be evenly queuing the extracts among the available processors. Most of the processes are 1-2 minutes but we have a few that are 10+ minutes so what we are seeing is most extracts complete but some only run after the long running extracts have completed rather than being processed by an available processor. When badly distributed, we sometimes end up with two 10+ minutes processes running consecutively

Is there any way to decide what load balancing technique to use? Or any other way to stop it distributing poorly?

Samy Abdul 3,376 Reputation points

2023-01-27T10:04:20.1733333+00:00

Hi @Ryan Abbey , the maximum limitation ,I could find in the documentation is 10,000 concurrent runs for all the pipelines and que limitation is

100

Concurrent pipeline runs per data factory that's shared among all pipelines in the factory 10,000 10,000
Ryan Abbey 1,186 Reputation points

2023-01-30T21:04:11.2933333+00:00

@Samy Abdul (Can't seem to respond direct to your comment)... if that were true, we would have fewer problems, we are definitely being limited to 20. Where did you get that detail from? Wondering if there's any way to determine why the discrepancy with what we are getting

Does anyone else get limited to 20?
Bhargava-MSFT 31,261 Reputation points Microsoft Employee Moderator

2023-02-02T21:56:52.1133333+00:00

Hello @Ryan Abbey,

It seems like @Samy Abdul is referring to the below document here.

https://github.com/MicrosoftDocs/azure-docs/blob/main/includes/azure-data-factory-limits.md
Ryan Abbey 1,186 Reputation points

2023-02-08T04:16:59.84+00:00

This would be the explanation of our limit

However, while good to note that we can have this increased to 50 (how do we do that?), it wasn't the basis of the question, it's the fact that pipelines appear to be assigned in a pre-execution round robin rather than having a pipeline assigned to an available process - this is causing delays when two long running pipelines are assigned to the same process (one having to wait for the other)
Bhargava-MSFT 31,261 Reputation points Microsoft Employee Moderator

2023-02-13T23:42:48.87+00:00

Hello @Ryan Abbey,

I am trying to find a way to increase the Max limit to 50 from my end, but it seems the limits need to increase from the backend.
Can you please submit a support case by selecting "service and subscription limits (quotas)"

Please let me know if you need any help with the support request.

Your answer

Samy Abdul 3,376 Reputation points

2023-01-27T10:04:20.1733333+00:00

Hi @Ryan Abbey , the maximum limitation ,I could find in the documentation is 10,000 concurrent runs for all the pipelines and que limitation is

100

Concurrent pipeline runs per data factory that's shared among all pipelines in the factory 10,000 10,000
Ryan Abbey 1,186 Reputation points

2023-01-30T21:04:11.2933333+00:00

@Samy Abdul (Can't seem to respond direct to your comment)... if that were true, we would have fewer problems, we are definitely being limited to 20. Where did you get that detail from? Wondering if there's any way to determine why the discrepancy with what we are getting

Does anyone else get limited to 20?
Bhargava-MSFT 31,261 Reputation points Microsoft Employee Moderator

2023-02-02T21:56:52.1133333+00:00

Hello @Ryan Abbey,

It seems like @Samy Abdul is referring to the below document here.

https://github.com/MicrosoftDocs/azure-docs/blob/main/includes/azure-data-factory-limits.md
Ryan Abbey 1,186 Reputation points

2023-02-08T04:16:59.84+00:00

This would be the explanation of our limit

However, while good to note that we can have this increased to 50 (how do we do that?), it wasn't the basis of the question, it's the fact that pipelines appear to be assigned in a pre-execution round robin rather than having a pipeline assigned to an available process - this is causing delays when two long running pipelines are assigned to the same process (one having to wait for the other)
Bhargava-MSFT 31,261 Reputation points Microsoft Employee Moderator

2023-02-13T23:42:48.87+00:00

Hello @Ryan Abbey,

I am trying to find a way to increase the Max limit to 50 from my end, but it seems the limits need to increase from the backend.
Can you please submit a support case by selecting "service and subscription limits (quotas)"

Please let me know if you need any help with the support request.

Share via

Dynamic load balance

Your answer