Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
2,872 questions
Synapse Managed Virtual Network Integration Runtime
Jeremy D. Busselman
16
Reputation points
Utilizing a Managed Virtual Network helps from a security aspect, and I am aware that the runtime needs to spin up.
What I'm struggling to understand or find documentation on is that, for EACH copy or dataflow step, needs to spin up the runtime, so if I have 3 copy steps in one pipeline, and it takes 2 minutes to spin up a runtime, then the overall duration of the pipeline has 6 minutes added, JUST to wait in queue to spin up. Imagine taking that by numerous pipelines.
Isn't there a way to spin up a runtime at the beginning of a batch cycle for all pipelines to use, so it's 2 minutes extra for the batch cycle, vs an exponential number based on spinning up for each and every step that needs it?
I can't imagine why anyone would choose this solution if you have to suffer the pain of this happening. There has to be something I'm missing??
Thanks
{count} votes
-
ShaikMaheer-MSFT 30,196 Reputation points Microsoft Employee2022-01-31T09:35:17.583+00:00 Hi @Jeremy D. Busselman ,
Thank you for posting query in Microsoft Q&A Platform.
We are checking this internally. I will share updates once I get updates from internal team.
@ShaikMaheer-MSFT I wanted to see if you've heard anything back on how to improve this issue?
Hi @Jeremy D. Busselman ,
Did you get chance to see if you can utilize TTL setting for IR. Below blog has more details about same. Kindly see if that helps.
https://techcommunity.microsoft.com/t5/azure-data-factory-blog/adf-adds-ttl-to-azure-ir-to-reduce-data-flow-activity-times/ba-p/878380
I have tried this but didn't seem to work. In reading the link above, it looks like it may ONLY work on the Data Flow tasks, and not any other pipeline tasks, along with only seeing the performance if there are multiple Data flow tasks within 1 pipeline, which is not what we have either.
Just to tag you back @ShaikMaheer-MSFT
Hi @Jeremy D. Busselman - yes spinning up cluster is the case with data flow activity only. For Copy activity or other activities there nothing called cluster spin up concept. I am still waiting for internal team to get updates on spin up runtime for Managed virtual Network IR. I will share updates soon.
Sign in to comment