Connecting from Synapse Managed private endpoint to an Application Gateway Private Link Configuration

Erwin Kramer 6

Is it possible to connect to an Application Gateway, which has a Private Link Configuration set (see https://learn.microsoft.com/en-us/azure/application-gateway/private-link-configure?tabs=portal), through a Synapse managed private endpoint? Currently, when selecting "New managed private endpoint (Private link service)" inside Synapse, you only have the option to select a dedicated Private Link Services resource, this type of resource is not something that a Private Link Configuration inside an Application Gateway generates or provides. I understand that this stuff is all new and some of it is in preview, but i like to know the possibilities and possible roadmap.

The use case is calling API's behind an API Management instance, which is positioned behind an Application Gateway, we want to figure out if this is a good way to create secure network connections (using the private IP address from our Application Gateway), we are aware that Self hosted runtimes are a possibility too for Synapse.

EDIT: Managed Private endpoints for Application Gateway work great now, I've posted a detailed article about it on https://github.com/erwinkramer/reference-architecture-secure-cross-tenant-API-traffic?tab=readme-ov-file#reference-implementation-for-microsoft-managed-origins . This should be the way to go forward for anyone struggling with secure cross-tenant connections.

HimanshuSinha-msft 19,381 Reputation points Microsoft Employee

2022-08-15T16:17:57.26+00:00

Hello @Erwin Kramer ,
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet .In case if you have any resolution please do share that same with the community as it can be helpful to others . Otherwise, will respond back with the more details and we will try to help .
Thanks
Himanshu
David Beavon 976 Reputation points

2022-08-16T19:52:13.577+00:00

@HimanshuSinha-msft @Vidya Narasimhan Hi Himanshu, The question wasn't really answered. Vidya gave a workaround, and probably didn't consider the scenarios that would use a REST API from Spark.

The original question wasn't specifically focus on copy-data pipelines (or integration runtimes). My scenario isn't really about pipelines either. We simply want our spark notebooks and unattended jobs to be able to reach private resources in our region.

FYI, It was possible to use REST API's in the Databricks deployment of Spark, because that service supports the injection of a normal vnet into the cluster when it is started. In Synapse, however, the "managed vnet" makes it overly difficult to reach an HTTPS API. We are highly dependent on Microsoft to provide us with their "managed private endpoint".

It would be helpful if you can share the possibilities and possible roadmap. This question came up in stackoverflow as well, and nobody seems to answer it for us.
Vidya Narasimhan 2,126 Reputation points Microsoft Employee

2022-08-17T15:21:26.203+00:00

Hi @Erwin Kramer , if your requirement is to call a REST API from Synapse Spark that is behind a managed VNET, one way you could achieve this is by using a similar architecture as described in this link https://learn.microsoft.com/en-us/azure/data-factory/tutorial-managed-virtual-network-on-premise-sql-server
You can replace ADF with Synapse and replace SQL server with your Rest endpoint in the architecture above. As described in the architecture, you need to provision private link service that is backed by NAT VMs inside the Forwarding VNet that forward requests to the backend API. In Synapse you can create the managed private endpoint for the private link service and provide the FQDN of the api as parameter.
Erwin Kramer 6 Reputation points

2022-08-17T15:30:58.337+00:00

Thanks for the suggestion. Doesn't seem much easier than running an SHIR, or would you disagree? Both are essentially VMs running permanently.
Vidya Narasimhan 2,126 Reputation points Microsoft Employee

2022-08-17T15:42:55.923+00:00

SHIR is not an option when using Spark behind a managed Vnet which is DEP enabled. Forgot to check with you if DEP is enabled.
The suggested way is not easy but possible approach for your scenario.
David Beavon 976 Reputation points

2022-08-17T18:26:41.84+00:00

@Vidya Narasimhan Thanks for the link related to load balancer and private link service. Have you ever tested that? We spent time looking into that; but it seemed very complex, and the TLS/HTTPS traffic was failing to jump thru all the required hoops.

The solution you pointed to requires the use of azure load balancer, and azure load balancer requires the use of a virtual machine in Azure. This solution isn’t a good one because of the protocol in use, HTTPS. The TLS connection needs to traverse some sort of transparent https proxy inside the virtual machine. At this point we’re getting super complicated. It is far more complex than what we could do with a simple vnet in databricks.

I had noticed that "application gateway" now has a private link that is in public preview. We also looked at using this as well ... but unfortunately there doesn't seem to be any support for the new private link in synapse or adf (ie. there is no "managed private endpoint" that can be configured to use the application gateway's private link configuration). Is there a roadmap for creating this managed private endpoint? How long does it typically take for ADF and Synapse to introduce a new managed private endpoint for something like this?

Ideally it wouldn't be so hard to make a API call from a spark cluster to a service which only uses private connectivity.
David Beavon 976 Reputation points

2022-08-17T18:29:24.16+00:00

We don't use DEP. We had tried it briefly but it was impossible to do basic things like AAD oauth2 authentication using Microsoft identities (Microsoft.Identity.Client).

The DEP seems like a good idea in principal, but it was too painful to use in practice.
Vidya Narasimhan 2,126 Reputation points Microsoft Employee

2022-08-18T14:26:34.647+00:00

We have some customers who have implemented the private link service with NAT VM architecture successfully.
I agree with you, it’s complex and any feature request from you is welcome.
David Beavon 976 Reputation points

2022-08-18T15:02:38.64+00:00

@Vidya Narasimhan Thanks for the reply.

FYI, We were not able to get this solution working, because of the TLS termination inside the VM. I'm guessing your proposal works well for SQL but not for Web API's.

Is there a way to see the existing "feature requests"?

Since I'm finding others asking for this on stackoverflow and here, it is difficult to believe that Microsoft is not already aware of the need for accessing API's privately from Spark clusters. (Web API's are fairly useful and common.) I'm guessing that the managed private endpoint for "application gateway" is already on the synapse backlog. I just wanted to know if that will take a month or a year or two years. It is so hard to get actionable communication out of this Synapse team.

In the meantime, are you aware of who I can talk to that might be able to get that private-link-service/load-balancer/nat-vm working? We are already in touch with some synapse pre-sales folks, and they have access to connect us with GBB's, but I need to know how to ask for the type of a GBB who will help us with this particular scenario. The one we spoke with before was unable to help.
Vidya Narasimhan 2,126 Reputation points Microsoft Employee

2022-08-19T14:52:54.773+00:00

Hi David, can you explain the issue with TLS termination better? Where are you doing teh TLS termination?
David Beavon 976 Reputation points

2022-08-19T16:46:51.673+00:00

@Vidya Narasimhan I copied that bit from a coworker, who says that the proposed solution would work fine for some protocols (SQL) but not for https traffic to a web API. I am not really very qualified to speak about network infrastructure, or about https traffic myself.

I can't say much except that I understood there were special considerations related to https traffic that didn't apply to the SQL Server wire protocol.

I have a regular pre-sales meeting with various Microsoft specialists (data and AI, synapse specialist, a GBB with pipeline experience in Synapse, etc). Typically these folks don't specialize in every topic, so they often put me in touch with someone with the required technical expertise. If you can tell me what kind of a networking expert we would need, I will ask them to find one.

I will also ask my coworker to explain our trouble using the proposed load-balancer solution for https traffic.

In the meantime I also wanted to focus on AG since that was the original topic of discussion. Is there a chance that Managed Private Endpoint for Application Gateway Private-Link will soon come to ADF & Synapse ? IE. Regardless of whether we are able to get the load-balancer/nat-vm working, it would (hopefully) be a temporary workaround that would go away after a more simple solution became available. It is hard for me to understand why managed private endpoints wouldn't be made available for ADF/Synapse at the same time as the AG private-link. Is there a lot of specialized and proprietary dev that is resource-specific?
Vidya Narasimhan 2,126 Reputation points Microsoft Employee

2022-08-21T07:28:28.063+00:00

Hi David,

Will wait for your co-worker to explain the issue.
AG private link is in public preview. Please create a feature request in the below link for support of AG private link in Synapse as managed private endpoint https://feedback.azure.com/d365community/forum/9b9ba8e4-0825-ec11-b6e6-000d3a4f07b8
David Beavon 976 Reputation points

2022-08-22T13:21:57.72+00:00

I added the idea for a managed private endpoint (https://feedback.azure.com/d365community/idea/5ba8ca7b-1c22-ed11-a81b-6045bd8606d4).

I also asked my coworker to explain the technical problems that prevent HTTPS traffic from traversing thru private-link-service/load-balancer/nat-vm.

In regards to the managed private endpoint for AG, is that likely to be a priority? Is it something that takes lots of specialized development effort? Is it just a configuration change to update the list of the managed private endpoints to include it? The reason I ask is because reaching API data from a spark cluster is pretty critical to our solutions. It would be helpful to know how long of a wait time will be involved (ie. is this type of thing going to take weeks or months or years?)
David Beavon 976 Reputation points

2022-08-23T01:25:55.887+00:00

@Vidya Narasimhan

My coworker said that the instructions are not written for the sake of HTTPS. I think if someone went to the trouble of explaining step-by-step how to make the solution work for HTTPS traffic, then that would make it more appealing to synapse customers.

Even so, this solution (traversing thru private-link-service/load-balancer/nat-vm) will be overly expensive, complex, and will require ongoing patching of the OS in the related VM. It is cumbersome because of the limitations of load-balancer. I think a solution that uses application-gateway would be a lot more appealing to the average synapse customer. Using HTTPS web services from a spark cluster should not be so hard!
Vidya Narasimhan 2,126 Reputation points Microsoft Employee

2022-08-23T04:25:40.937+00:00

Hi David, if you can please go through the IPtables rules setup of NAT VM in detail in the doc link, you would see that the NAT VMs are just doing IP/Port forwarding using TCP. So the solution is not specific to SQL Server. You can point to a web/api server by providing the right IP/Port for destination instead of SQL server and forward the request using the TCP rule in specified in iptables, Details of setting up the IPTable rules are well described in the same doc.
Alonso Roa, Walter Javier 6 Reputation points

2023-02-16T00:15:55.06+00:00

Hi @Vidya Narasimhan

I have a similar scenario, basically we want Synapse doing an API request to an app service, this app service has access restriction enabled, do you have any guidance to be able to allow connectivity from Synapse to app service?
David Beavon 976 Reputation points

2023-05-27T18:17:09.06+00:00

Hi @Vidya Narasimhan Sorry to keep pulling you back into this discussion but I think the whole NAT VM thing is a little intimidating for my team (I'm more motivated to using it than they are). The problem is not only the NAT VM but all the other pieces as well (load balancer and private link service).

I think the Application Gateway, with its Private Link Configuration is the approach that would appeal to the most people. It is a level 7 load balancer that is more approachable and has slightly easier configuration.

Given that synapse (and adf) managed vnets are still not supporting Application Gateway in their U/I, I am wondering if there is a back-door way to route our traffic to that private link. Do you think the REST API for Synapse should allow us to create the MPE?

https://learn.microsoft.com/en-us/rest/api/synapse/data-plane/managed-private-endpoints/create?tabs=HTTP

https://stackoverflow.com/questions/73389174/azure-devops-pipeline-create-a-synapse-managed-private-endpoints-to-a-azure-st

Any comments would be appreciated. Also, if you have any insights on why these teams (Synapse and ADF) have not prioritized application gateway, then that would be helpful to understand as well. It seems like application gateway would be a lot more popular than it is.
David Beavon 976 Reputation points

2023-06-14T15:09:07.8566667+00:00

Not sure if everyone is aware but "private link" for application gateway is finally generally available.

I'm hoping that means we will soon be able to use it from a managed vnet in synapse or adf. Unfortunately as-of today (6/14/2023) the U/I doesn't provide this as a selection within the list of available MPE's.

Given that this discussion here is a year old, and has gotten a bit noisy, and given the "GA" of "private link", I'm re-opening the discussion again to get a fresh start in stack-overflow:
https://stackoverflow.com/questions/76469457/how-do-i-create-a-managed-private-endpoint-in-synapse-and-adf-that-will-allow

Title is "How do I create a managed private endpoint in Synapse (and ADF) that will allow me to connect to data via application gateway"
Erwin Kramer 6 Reputation points

2024-05-15T20:27:34.0733333+00:00

Managed Private endpoints for Application Gateway work great now, I've posted a detailed article about it on https://github.com/erwinkramer/reference-architecture-secure-cross-tenant-API-traffic?tab=readme-ov-file#reference-implementation-for-microsoft-managed-origins

This should be the way to go forward for anyone struggling with secure cross-tenant connections.

2 answers

Vidya Narasimhan 2,126 Reputation points Microsoft Employee

2022-07-29T10:27:39.467+00:00

@Erwin Kramer , a managed private endpoint for a private link service is not useful for your scenario as private link service can be enabled only for services running behind a load balancer which is not true in your case.
The recommended way is to use Synapse SHIR that has line of sight to your app gateway private IP.
Please sign in to rate this answer.

0 comments No comments
Sign in to comment
Erwin Kramer 6 Reputation points

2024-05-15T20:25:12.7+00:00

Managed Private endpoints for Application Gateway work great now, I've posted a detailed article about it on https://github.com/erwinkramer/reference-architecture-secure-cross-tenant-API-traffic?tab=readme-ov-file#reference-implementation-for-microsoft-managed-origins

This should be the way to go forward for anyone struggling with secure cross-tenant connections.
Please sign in to rate this answer.

0 comments No comments
Sign in to comment

Share via

Connecting from Synapse Managed private endpoint to an Application Gateway Private Link Configuration

2 answers