Azure Synapse Managed VNET and connectivity to the serverless endpoint issues from a pipeline

colinrippeyfinarne 26 Reputation points
2021-10-04T09:58:30.5+00:00

I have a scenario that (I'm sure) a lot of folk will want to use but I am running into issues inside a Synapse workspace that uses a Managed VNET.

If we can take a simple Synapse workspace that is NOT on a Managed VNET first.

I create a Linked Service inside synapse that I configure as an Azure SQL, I use the current serverless endpoint as the fqdn and connect via Managed Identity. At this point click Test Connection it works.

I then create an integration dataset that "points" to the new linked service, the master database.

I then create an integration dataset of type Parquet and point to a new folder on the connected data lake.

I then create a Pipeline and add the Copy Data, for the source I use the integration dataset and use a simple query say

SELECT * from sys.databases

I set the sink as the Parquet dataset.

This is a very simple "can I execute a query against the current serverless endpoint and get back results as part of a pipeline".

We have been using this connection quite a lot, we have a ton of data in our lake we expose using the serverless endpoint to external consumers such as Power BI etc. and we have also written some pipelines that consume the serverless endpoint as if it was an Azure SQL database.

Every part of the connectivity/debug experience all works from Test Connection to Preview Data. If I run the pipeline I get a Parquet file created so all good.

If I try this in a Synapse workspace that uses a Managed VNET I get no issues with Test Connection, Preview Data, but when I try to run the pipeline I see the pipeline run and attempts to connect to the SQL linked service but never succeeds (basically the pipeline sits for minutes trying to connect to the serverless endpoint and I then cancel it).

So I am missing something in terms of how the integration runtimes connect to the serverless endpoints on a Managed VNET it seems.

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
4,402 questions
{count} votes