Synapse Managed Virtual Network Integration Runtime

Jeremy D. Busselman 36

Utilizing a Managed Virtual Network helps from a security aspect, and I am aware that the runtime needs to spin up.

What I'm struggling to understand or find documentation on is that, for EACH copy or dataflow step, needs to spin up the runtime, so if I have 3 copy steps in one pipeline, and it takes 2 minutes to spin up a runtime, then the overall duration of the pipeline has 6 minutes added, JUST to wait in queue to spin up. Imagine taking that by numerous pipelines.

Isn't there a way to spin up a runtime at the beginning of a batch cycle for all pipelines to use, so it's 2 minutes extra for the batch cycle, vs an exponential number based on spinning up for each and every step that needs it?

I can't imagine why anyone would choose this solution if you have to suffer the pain of this happening. There has to be something I'm missing??

Thanks

ShaikMaheer-MSFT 38,401 Reputation points Microsoft Employee

2022-01-31T09:35:17.583+00:00

Hi @Jeremy D. Busselman ,

Thank you for posting query in Microsoft Q&A Platform.

We are checking this internally. I will share updates once I get updates from internal team.
Jeremy D. Busselman 36 Reputation points

2022-02-10T19:04:00.73+00:00

@ShaikMaheer-MSFT I wanted to see if you've heard anything back on how to improve this issue?
ShaikMaheer-MSFT 38,401 Reputation points Microsoft Employee

2022-02-15T16:32:37.927+00:00

Hi @Jeremy D. Busselman ,

Did you get chance to see if you can utilize TTL setting for IR. Below blog has more details about same. Kindly see if that helps.
https://techcommunity.microsoft.com/t5/azure-data-factory-blog/adf-adds-ttl-to-azure-ir-to-reduce-data-flow-activity-times/ba-p/878380
Jeremy D. Busselman 36 Reputation points

2022-02-15T17:07:40.317+00:00

I have tried this but didn't seem to work. In reading the link above, it looks like it may ONLY work on the Data Flow tasks, and not any other pipeline tasks, along with only seeing the performance if there are multiple Data flow tasks within 1 pipeline, which is not what we have either.
Jeremy D. Busselman 36 Reputation points

2022-02-15T17:08:13.703+00:00

Just to tag you back @ShaikMaheer-MSFT
ShaikMaheer-MSFT 38,401 Reputation points Microsoft Employee

2022-02-15T17:40:56.93+00:00

Hi @Jeremy D. Busselman - yes spinning up cluster is the case with data flow activity only. For Copy activity or other activities there nothing called cluster spin up concept. I am still waiting for internal team to get updates on spin up runtime for Managed virtual Network IR. I will share updates soon.

Share via

Synapse Managed Virtual Network Integration Runtime