How to setup DNS to use private endpoints for Data Factory portal with multiple environments?

Question

How to setup DNS to use private endpoints for Data Factory portal with multiple environments?

Paul Hernandez 686 Microsoft Employee

How do we need to setup DNS so that we can use private endpoints for Data Factory portal with multiple environments ?

Data Factory private endpoint a non-unique FQDN portal.adf.azure.com

This is creating a problem since we are using Hub - Spoke network topology and all the Private DNS zones are linked to the HUB network

It's not possible to:

Link multiple Private DNS zones with the same name adf.azure.com to HUB network
Create multiple DNS records for portal.adf.azure.com in a single DNS zone

A diagram of the current solution:

User's image

Answer accepted by question author

0 additional answers

Your answer

Answer 1

Anonymous

@Paul Hernandez

Thanks for Using Microsoft Q&A platform.

it is not possible to link multiple Private DNS zones with the same name adf.azure.com to HUB network or create multiple DNS records for portal.adf.azure.com in a single DNS zone. Currently, there is only one Data Factory portal endpoint, so there is only one private endpoint for the portal in a DNS zone.

One alternative is to remove the private endpoint and DNS configuration.

Another option is to use a managed virtual network in private endpoints in Azure Data Factory.

However, it is important to ensure that the DNS settings are correct when using the fully qualified domain name (FQDN) for the connection, and the settings must resolve to the private IP address of the private endpoint.

for references: Azure Private Link for Azure Data Factory - Azure Data Factory | Microsoft Learn

If you have further queries, please do let us know.

Paul Hernandez 686 Reputation points Microsoft Employee

2023-10-14T06:59:29.98+00:00

Hi @Anonymous , Thanks for your input. I'll digest the proposed workaround and test it. The reference link you've shared seems to be broken. Could you please check it?
Paul Hernandez 686 Reputation points Microsoft Employee

2023-10-14T14:15:50.8566667+00:00

Hi @Anonymous , Could you please elaborate a bit more on this "Another option is to use a managed virtual network in private endpoints in Azure Data Factory." Could you please share a diagram representing this option?
Anonymous

2023-10-16T04:56:18.02+00:00
@Paul Hernandez

Creating an integration runtime within a managed virtual network ensures the data integration process is isolated and secure. With a managed virtual network, you can offload the burden of managing the virtual network to Data Factory. You don't need to create a subnet for an integration runtime that could eventually use many private IPs from your virtual network and would require prior network infrastructure planning. Deep Azure networking knowledge isn't required to do data integrations securely. Instead, getting started with secure ETL is much simpler for data engineers. A managed virtual network along with managed private endpoints protects against data exfiltration.

Here is a diagram that shows the architecture of a Data Factory managed virtual network:

As you can see in the diagram, the managed virtual network is created within the data factory, and the integration runtime is provisioned within the managed virtual network. The integration runtime uses private endpoints to securely connect to supported data stores.

To use private endpoints with a managed virtual network in Azure Data Factory, you need to follow these steps:

Create a managed virtual network in your data factory.

Create an integration runtime within the managed virtual network.

Configure private endpoints for the supported data stores that you want to connect to.

For more information on how to create a managed virtual network and configure private endpoints in Azure Data Factory, please refer to the following documentation:

Managed virtual network and managed private endpoints in Azure Data Factory

Create a managed virtual network in Azure Data Factory

Configure private endpoints for Azure Data Factory

I hope this helps! Let me know if you have any further questions.
Anonymous

2023-10-17T08:52:58.71+00:00

@Paul Hernandez We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
Paul Hernandez 686 Reputation points Microsoft Employee

2023-10-17T09:06:33.3433333+00:00

Hi @Anonymous , thanks for the follow-up.

We accepted this as a current limitation:

And will continue with the hub-spoke architecture.

We are facing a similar situation with Azure Databricks and Purview.
Anonymous

2023-10-18T14:05:50.8266667+00:00

@Paul Hernandez

could you please confirm us that you are satisfied with present situation?

if not please more details of your requirement .
Paul Hernandez 686 Reputation points Microsoft Employee

2023-10-18T18:08:07.22+00:00

I'm satisfied with the support you provided @Anonymous thanks
Anonymous

2023-10-19T09:07:13.8366667+00:00

@Paul Hernandez Hope this helps. Do let us know if you any further queries.

please, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.
Anonymous

2023-10-19T11:15:32.9733333+00:00

@Paul Hernandez

thanks for accepting the answer.

Share via

How to setup DNS to use private endpoints for Data Factory portal with multiple environments?

0 additional answers

Your answer