Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
5,331 questions
Synapse Managed Virtual Network Integration Runtime
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(313.6, 69%, 40%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJD%3C/text%3E%3C/svg%3E)
Jeremy D. Busselman
36
Reputation points
Utilizing a Managed Virtual Network helps from a security aspect, and I am aware that the runtime needs to spin up.
What I'm struggling to understand or find documentation on is that, for EACH copy or dataflow step, needs to spin up the runtime, so if I have 3 copy steps in one pipeline, and it takes 2 minutes to spin up a runtime, then the overall duration of the pipeline has 6 minutes added, JUST to wait in queue to spin up. Imagine taking that by numerous pipelines.
Isn't there a way to spin up a runtime at the beginning of a batch cycle for all pipelines to use, so it's 2 minutes extra for the batch cycle, vs an exponential number based on spinning up for each and every step that needs it?
I can't imagine why anyone would choose this solution if you have to suffer the pain of this happening. There has to be something I'm missing??
Thanks
Azure Synapse Analytics
{count} votes
-
ShaikMaheer-MSFT 38,531 Reputation points Microsoft Employee Moderator2022-01-31T09:35:17.583+00:00 Hi @Jeremy D. Busselman ,
Thank you for posting query in Microsoft Q&A Platform.
We are checking this internally. I will share updates once I get updates from internal team.
-
Jeremy D. Busselman 36 Reputation points
2022-02-10T19:04:00.73+00:00 @ShaikMaheer-MSFT I wanted to see if you've heard anything back on how to improve this issue?
-
ShaikMaheer-MSFT 38,531 Reputation points Microsoft Employee Moderator
2022-02-15T16:32:37.927+00:00 Hi @Jeremy D. Busselman ,
Did you get chance to see if you can utilize TTL setting for IR. Below blog has more details about same. Kindly see if that helps.
https://techcommunity.microsoft.com/t5/azure-data-factory-blog/adf-adds-ttl-to-azure-ir-to-reduce-data-flow-activity-times/ba-p/878380 -
Jeremy D. Busselman 36 Reputation points
2022-02-15T17:07:40.317+00:00 I have tried this but didn't seem to work. In reading the link above, it looks like it may ONLY work on the Data Flow tasks, and not any other pipeline tasks, along with only seeing the performance if there are multiple Data flow tasks within 1 pipeline, which is not what we have either.
-
Jeremy D. Busselman 36 Reputation points
2022-02-15T17:08:13.703+00:00 Just to tag you back @ShaikMaheer-MSFT
-
ShaikMaheer-MSFT 38,531 Reputation points Microsoft Employee Moderator
2022-02-15T17:40:56.93+00:00 Hi @Jeremy D. Busselman - yes spinning up cluster is the case with data flow activity only. For Copy activity or other activities there nothing called cluster spin up concept. I am still waiting for internal team to get updates on spin up runtime for Managed virtual Network IR. I will share updates soon.
Sign in to comment
Sign in to answer