Is the concurrency control setting in synapse pipeline broken?

Question

Can the concurrency setting in Synapse Pipeline help control the number of concurrent runs of a given pipeline?

Does this configuration work even when orchestrating pipeline run from another pipeline?

In a separate thread here, it is advised that concurrency setting can be used to restrict multiple instances of a pipeline running. But, when trying to use that in the context of a parent-child pipeline, I am finding the behaviour not exactly as documented.

My synapse pipeline setup is like-

parent pipeline A -> invokes -> child pipeline ZA

parent pipeline B -> invokes -> child pipeline ZA

parent pipeline C -> invokes -> child pipeline ZA

I wanted to restrict the number of concurrent runs of the pipeline ZA to 2 so I have configured the concurrency setting on pipeline ZA to 2. I was expecting that when all the three parent pipelines get triggered at the same time, only two of the ZA invocations will get started and the third one will wait in the queue.

Instead, what I am finding is that, all three runs of the pipeline ZA initiate completely ignoring the concurrency setting set by me.

Is this a bug in the synapse pipeline? Should I not see the expected behaviour as I described above? The documentation is very vague in this regard.

Answer 1

@DhruvinGajjar-8294 Welcome to Microsoft Q&A forum and thanks for providing the detailed explanation of the issue you are experiencing.

I just did a quick test with both Synapse and ADF pipelines with same requirement you have shared and noticed that the concurrency setting is behaving as expected.

Here is my child pipeline concurrency setting:

Child pipeline JSON payload used for testing:

{
    "name": "pl_Child_ZA",
    "properties": {
        "activities": [
            {
                "name": "Wait1",
                "type": "Wait",
                "dependsOn": [],
                "userProperties": [],
                "typeProperties": {
                    "waitTimeInSeconds": 25
                }
            }
        ],
        "concurrency": 2,
        "annotations": []
    }
}

As expected, when I triggered all three parent pipelines which are supposed to execute the same child pipeline, out of three executions, first 2 child pipeline runs are in-progress and the 3rd run is Queued which is expected as the concurrency on it is set to 2.

Below are the 3 parent pipeline JSON payloads used for testing:

pl_Parent_ZA1:

{
    "name": "pl_Parent_ZA1",
    "properties": {
        "activities": [
            {
                "name": "Execute Pipeline1",
                "type": "ExecutePipeline",
                "dependsOn": [],
                "userProperties": [],
                "typeProperties": {
                    "pipeline": {
                        "referenceName": "pl_Child_ZA",
                        "type": "PipelineReference"
                    },
                    "waitOnCompletion": true
                }
            }
        ],
        "annotations": []
    }
pl_Parent_ZA2

pl_Parent_ZA2:

{
    "name": "pl_Parent_ZA2",
    "properties": {
        "activities": [
            {
                "name": "Execute Pipeline1",
                "type": "ExecutePipeline",
                "dependsOn": [],
                "userProperties": [],
                "typeProperties": {
                    "pipeline": {
                        "referenceName": "pl_Child_ZA",
                        "type": "PipelineReference"
                    },
                    "waitOnCompletion": true
                }
            }
        ],
        "annotations": []
    }
}

pl_Parent_ZA3:

{
    "name": "pl_Parent_ZA3",
    "properties": {
        "activities": [
            {
                "name": "Execute Pipeline1",
                "type": "ExecutePipeline",
                "dependsOn": [],
                "userProperties": [],
                "typeProperties": {
                    "pipeline": {
                        "referenceName": "pl_Child_ZA",
                        "type": "PipelineReference"
                    },
                    "waitOnCompletion": true
                }
            }
        ],
        "annotations": []
    }
}

Feel free to use the above templates to test it on your end and let us know if you still notice the same behavior.

If you continue to notice the same behavior with your pipeline, I request you to please share screenshot of your published child pipeline settings, images from Monitor tab showing that all three child pipelines are in-progress at same time irrespective of concurrency setting.

Please ensure to publish the pipeline after setting the concurrency.

Hope this info helps.

---

Please don’t forget to Accept Answer and Yes for "was this answer helpful" wherever the information provided helps you, this can be beneficial to other community members.