What is the reason for networking problems in ADF "managed vnet"?

David Beavon 976

I have used "managed vnets" in a few different scenarios. On one hand, there is a managed vnet in Power BI, used by the "Power BI Managed VNET Gateway". They say their technology originated from within ADF, which uses managed vnets as well. Within ADF they are used for integration runtimes.

Another new place I have used the "managed vnet" technology is in Synapse Analytics. In this context the technology is used for the sake of hosting VM's that participate in a spark cluster.

In all the three scenarios where I've been exposed to Microsoft "managed vnets", they have been extremely buggy and unreliable. For example, the bugs in Power BI Managed VNET Gateway cause refresh operations to fail on a recurring basis (about 50% chance of failure each day). The related support ticket has been open for two years. And they are definitely blaming the gateway failures on the "network team". Similarly in ADF these network bugs in the VNET will force us to enable retries to repeat operations over and over until they succeed (thereby incurring a large cost to our subscription).

In Synapse Analytics the buggy VNET causes failures as well. The failures take place in my own custom code (Spark executors) and Microsoft components. The Microsoft Livy Jobs Service (a component of Synapse itself) is very unreliable. Below is a screenshot of my Spark jobs from two days ago. Notice the failures labeled #1 and #2.

User's image

The section labeled #1 are failures in my own custom code. The failures are network exceptions, consisting of disconnections with a message such as "connection reset by peer". They might be related to disconnections from a remote HTTP service or SQL database or identity server.

The section labeled #2 are failures of Synapse to submit new jobs. These are self-contained failures that have nothing to do with my custom code. Basically the Spark PG is relying on an "LSR" operation that is serviced by the ADF PG. This section of the failures does not involve any of my own custom code. It is a problem where Microsoft components are failing because of the unreliable "managed vnet".

Can someone please help me understand this "managed vnet" technology? What is the reason for all the reliability problems, in the three different places where I have seen it being used used? Is it failing under high load? Are there software components involved, which might be starved for CPU?

ShaikMaheer-MSFT 38,406 Reputation points Microsoft Employee

2023-10-16T06:57:32.0166667+00:00

Hi David Beavon,

Thank you for posting query in Microsoft Q&A Platform.

Managed Virtual Network (VNet) is a feature in Azure that allows you to create a virtual network that is managed by Azure. This means that Azure takes care of the underlying infrastructure, such as the network topology, routing, and security, while you focus on configuring your applications and services.

The Managed VNet technology is used in various Azure services, including Power BI, Azure Data Factory, and Azure Synapse Analytics, to provide secure and reliable connectivity between different components of these services. However, as with any technology, there can be issues that arise, especially under high load or when there are software components involved.

Some of the common issues that can arise with Managed VNet include:

Network connectivity issues: This can happen if there are issues with the underlying network infrastructure, such as firewalls or routing issues.

Resource constraints: If the resources allocated to the Managed VNet are insufficient, it can lead to performance issues or failures.

Software bugs: As with any software, there can be bugs or issues that arise, especially under high load or when there are complex interactions between different components.

To troubleshoot the issues you are experiencing with Managed VNet, you may need to work with Microsoft Azure Support to identify the root cause of the issues and determine the best course of action to resolve them. This may involve analyzing logs, monitoring performance metrics, and testing different configurations to identify the root cause of the issues.

In general, it is important to ensure that you have allocated sufficient resources to the Managed VNet, such as CPU, memory, and network bandwidth, to ensure that it can handle the load placed on it. You should also monitor the performance of the Managed VNet and its components to identify any issues before they become critical. Finally, it is important to stay up-to-date with the latest patches and updates for the Managed VNet and its components to ensure that any known issues are addressed.
David Beavon 976 Reputation points

2023-10-16T13:30:22.64+00:00

@ShaikMaheer-MSFT Thanks for the response, it covered too much ground and was not especially helpful. (except, perhaps, if you audience had never worked with Managed VNETs in the past). You can see from my question that I've worked with these VNETs frequently, and still do not understand why they are so failure-prone. Apparently neither do you.

Can you confirm that you are not a bot, and then please elaborate on the part of your post where you mention "Resource constraints". What are the resources that are potentially constrained in a Microsoft managed vnet? Are they the types of things that will fail under high load? Is it something easily understood by a customer, like memory, CPU, or ephemeral TCP ports?

You can also see in my post that I have already engaged Azure Support and have an ongoing Power BI case for two years. I'm not sure why you are asking me to work with Azure support when I'm already doing so. If Microsoft is allowing bots to reply to the posts in this forum, then the quality of the Q&A is going to deteriorate very rapidly. I'm not looking for these types of answers; I'm looking for help from human beings who have first-hand experiences.
David Beavon 976 Reputation points

2023-10-16T13:33:11.8766667+00:00

Can you elaborate on the resources used in workings of the Managed vnets themselves?

You mentioned the possibility of "resource constraints" and that seems to be the most useful part of your post.
ShaikMaheer-MSFT 38,406 Reputation points Microsoft Employee

2023-10-20T17:41:48.41+00:00

Hi David Beavon, Managed Vnets usually will be handled by Azure Platform by itself. So if there is any behind the scene resource constraints that causing issue, then it might need to check by support team only.
As you have mentioned that, you have already created a support ticket. I believe support team would be helpful in this case to debug deeper and understand what causing issue in your specific case. Feel free to share your support ticket number here. I will also see if any push i can from my end on that ticket. Thank you.
David Beavon 976 Reputation points

2023-10-20T23:36:43.5433333+00:00

@ShaikMaheer-MSFT

Thanks for the response. I can see now you are not a bot after all!

What is the resource constraint? My understanding is that the "managed vnet" is implemented as three layers of complex virtualization. (1) actual hardware, (2) virtual machines used for load balancing and fault-tolerance, and (3) docker containerization - within the virtual machines - that is used to try and isolate the vnet traffic for the various customers/tenants.

Oddly enough I've heard that multiple customers may share the same VM's, and these aren't dedicated to one customer at a time. Perhaps this was done to improve the startup/shutdown times of these "managed vnets". Or maybe it was just less expensive that way.

These three layers of virtualization may result in three possible points of failure. I'm guessing that it takes some tricks for the TCP network activity to be coordinated between all three layers, before it is transmitted out over physical infrastructure. I wish I was able to troubleshoot myself. The trade-off for using a "managed vnet" is that this is secret sauce which customers cannot troubleshoot on our own, even if we wanted to.

As far as resource constraints go, perhaps there is some sort of bootstrapping failures that happen at the container level, and the customer might see transient failures for a little while before everything is stabilized. Even this doesn't seem like a good excuse. The managed vnet's network interfaces should not be actively used until some sort of health checkup can be completed on it.

I opened my first support case many years ago with the Power BI team. And another second case is underway now as well (for about a month ). The professional support ticket is #2310130040012067. The date on that case is a bit misleading - I had already been working on it prior to creating this ticket.

This CSS team (ADF) seems to be unwilling to disclose any more logs or exception details for the current incident (a twenty minute outage). It is odd that they are being so secretive with me now, especially since I was already able to see these logs and error messages in the prior incidences of the same bug. When I first had a chance to look at it, the underlying exception that was observed in the LSR seems to be a very basic socket error ("connection reset by peer"). That exception by itself isn't actually interesting. What is much more interesting is the fact that it won't go away for twenty minutes!!!

I'm currently trying to transfer the case to Microsoft. So far the only people engaged in working on it are third-party support engineers (CSS support engineers at Mindtree, without any PTA assistance).

There is already an ICM# and a bug# as well. Unfortunately the bug that the ADF PG described is NOT actually going to address the underlying source of my problems (the socket errors and network outages). That bug only appears to change the way the socket error is reported when two Microsoft components interact - LSR and JOBS-SERVICE. So my fear is that Microsoft is trying to use some sort of a workaround that will serve the purpose of the JOBS-SERVICE, but it will NOT actually try to help me address the networking outages themselves. My custom code is also significantly impacted by the networking outages in the "managed vnet". I'm at a loss on how to work around the network outages. There isn't any way for me to accommodate a twenty minute outage, no matter how many "retries" I try to sprinkle around.

Hopefully it doesn't sound too cynical, but I believe these problems in the "managed vnet" are already familiar to the ADF teams (both CSS and PG). When it comes to "ADF pipelines", these folks are very quick to tell customers to "enable retries". This recommendation serves their purposes and it shifts the responsibility back to the customer (and some of the costs as well!) Unfortunately the recommendation doesn't work well in other parts of Synapse (like for Spark batches, and job scheduling). Interestingly there are some other areas of Synapse where they have decided NOT to use the "managed vnets". For example, I believe that the team responsible for SQL Dedicated Pools has decided to avoid this technology. I'm guessing it is because it is still experimental and pretty unreliable.

I have heard rumors that some of the bugs may be fixed in 2024 or 2025 but the exact date is not well publicized. And certain CSS teams don't appear to be at liberty to share an ETA, even when a customer is directly impacted by these vnet problems.

I have also heard that there is a dedicated CSS team that works on "managed vnet" cases, and it works in proximity to the ADF team. But my ticket hasn't engaged with them yet.

Any help would be very much appreciated.
ShaikMaheer-MSFT 38,406 Reputation points Microsoft Employee

2023-10-23T07:47:31.0066667+00:00

Hi David Beavon, Apologies for all the trouble you are facing. I am trying to see if I can push from my end on this above ticket and see if correct team engages and do right things to help you here. Thank you.
David Beavon 976 Reputation points

2023-10-24T23:20:14.5566667+00:00

@ShaikMaheer-MSFT

Anything you can do would be greatly appreciated. There doesn't seem to be an urgency to work on my support case, despite ongoing outages and failures. I would have hoped that twenty minutes of unexpected errors on my Synapse workspace would be of concern to someone (either on the Microsoft or the Mindtree side of things).

... I suspect there are multiple teams engaged on the CSS side and also on the PG side. Once there are more than two or three teams have become involved in a case, then these support cases can become bogged down in a very substantial way. I suspect the delays are from both the Microsoft side and the Mindtree side. Any pushing from your end might save me a couple of months of unnecessary delays. You can tell that I am having quite a bit of trouble finding the right channels to get support! (Otherwise I wouldn't be reaching out to the entire Internet.)
David Beavon 976 Reputation points

2023-10-31T23:08:00.2566667+00:00

@ShaikMaheer-MSFT were you able to push this?

1 answer

David Beavon 976 Reputation points

2023-11-15T23:33:23.17+00:00
CAVEAT: This is not intended to be extremely reliable information and I cannot give a source for now. Anyone finding this information should keep in mind that the context is related to the Power BI managed VNET gateway. I don't have any certainty that this information extends to other platforms that use "managed vnets" (eg ADF and Synapse).

On the Power BI Gateway side of things, I have confirmation of two changes coming in 2024 that might make an improvement. See below. These changes are for the "Azure Network stack"

Feature ending with x509 : Support map space deletion in VFP without breaking flows.

(virtual filtering platform, ie switch?)

The ETA publicized is late 2024

Bug ending with x810 : PE-NC Flow mix up

Network team promised to roll it out early 2024.

... Please note that the Power BI Managed VNET Gateway is still in preview, so it is reasonable that there are ongoing changes that would improve reliability. (The only unreasonable part is waiting 3 or 4 years for the GA.)

In contrast to Power BI, I am not having as much luck with some other PG teams. I'm still waiting on a couple other PG teams to explain their managed vnet problems as well (eg. ADF and Synapse). This may take a while. I have made zero progress in the past month, despite spending dozens of hours with the ADF/Synapse organizations at CSS. Any information that I am able to gather by way of CSS is very hard to come by! CSS is quite different than unified support, to put it lightly. Your Azure Account Manager will be very eager to point you to one of these, but not the other.
Please sign in to rate this answer.
David Beavon 976 Reputation points

2023-11-17T01:54:31.5333333+00:00

Just to be more precise about the name of the related Azure platform team .. I've heard that the team is known as the Azure Network SDN Team (Software Defined Networking).

I'm definitely looking forward to their bug fixes in 2024. I really wish that the team would share more publicly about their roadmaps, since I have been struggling for so many years with these virtualized network components! Especially in Synapse/Spark pools. Rest assured, all of my CSS support cases will remain open until these bugs are fixed and the errors stop happening. It is tiring. I have seen enough socket exceptions to last me the rest of my life!

David Beavon 976 Reputation points

2023-11-17T01:56:21.79+00:00

Tip to those working with CSS on these vnet network issues. After about a month, you should push harder on the ops manager to engage with a PTA. Especially if the PG and CSS teams don't seem to have any reliable communication channels.
Sign in to comment

Share via

What is the reason for networking problems in ADF "managed vnet"?

1 answer