How to setup Azure databricks with pubic access disabled and configuring the private DNS for SSO login?

Question

How to setup Azure databricks with pubic access disabled and configuring the private DNS for SSO login?

Sunilprasath Elangovan 60

Hi Team,

Good afternoon.

We are trying to setup Azure Databricks with public access disabled, by following this page.

https://learn.microsoft.com/en-us/azure/databricks/administration-guide/cloud-configurations/azure/private-link-standard

We followed all the steps ( ex dedicative private workspace for browser authentication ), but after setting up this , still we are directed to public ip address to authenticate.

All the private endpoints (front,back,browser authentication), private DNS configuration we have done in our Hub Subscription VNET ,which is connected with our on-premises network via Site-to-Site VPN. Users login to Azure databricks from that on-premises network.

User's image

we followed the similar setup for key vault ,, and we are able to access them from on-prem using the private IP address.

Screenshot 2023-05-02 2205555

Thanks for your answers in advance.

Regards,

Sunil

PRADEEPCHEEKATLA 90,611 Reputation points

2023-05-05T04:30:19.8633333+00:00

@Sunilprasath Elangovan - Thanks for the question and using MS Q&A platform.

We are reaching out to the internal team to get more information related to your query and will get back to you as soon as we have an update.
PRADEEPCHEEKATLA 90,611 Reputation points

2023-05-15T05:27:24.2966667+00:00

@Sunilprasath Elangovan - Just checking in to see if the below answer helped. If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.

Accepted answer

0 additional answers

Your answer

PRADEEPCHEEKATLA 90,611 Reputation points

2023-05-05T04:30:19.8633333+00:00

@Sunilprasath Elangovan - Thanks for the question and using MS Q&A platform.

We are reaching out to the internal team to get more information related to your query and will get back to you as soon as we have an update.
PRADEEPCHEEKATLA 90,611 Reputation points

2023-05-15T05:27:24.2966667+00:00

@Sunilprasath Elangovan - Just checking in to see if the below answer helped. If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.

Answer 1

PRADEEPCHEEKATLA 90,611

@Sunilprasath Elangovan - Thanks for the question and using MS Q&A platform.

From your diagram, it seems like frontend and backend private links are in the same vnet, which makes no sense in the same subnet since they will point out to the same resource.

You can check in the private dns zone if there are the proper A records in there, as per the documentation you refer.

But I think the problem is related to a more architectural design, in the documentation you refer, there isn't any vnet peering, transit and customer data plane vnets are not peered and have different dns zones.

Normally in this type of hub & spoke network topologies, with vnet peering, you would have 3 vnets:

Transit vnet: Has vpn gateway and private endpoint for web auth, linked with dns zone
Hub vnet: Has private endpoint for web app and storage accounts
Data Plane vnet: linked with dns zone

Peering happens between transit and hub, and between hub and data plane.

Hope this helps. Do let us know if you any further queries.

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.

Sunilprasath Elangovan 60 Reputation points

2023-05-08T13:45:54.1533333+00:00

@PRADEEPCHEEKATLA , Thanks for your response.

Apologize for not making clear diagram (missed to include the vnet)

Currently we dont have 3 vnets , we have only HUB vnet (which also acts as Transit Vnet , it has VPN gateway connectivity with on-oprem) and SPOKE vnet where we deployed our services Storage account, key vault and databricks.

So kindly guide us , where we have to deploy the following private endpoints.

frontend --> ? dedicative subnet in SPOKE Vnet? or can we use the subnet where we deployed our keyvault private endpoint

backend --> ? dedicative subnet in HUB vnet?

browser authentication --> ? dedicative subnet in HUB vnet?

Can we add the private IPs of those private endpoints in the private DNS zone in HUB ?

Let me know if you have any questions
PRADEEPCHEEKATLA 90,611 Reputation points

2023-05-10T08:27:55.8833333+00:00

@Sunilprasath Elangovan - I'll try to provide clarity instead of directly answering your questions, so you can take an informed decision on where to place them according with your requirements.

Frontend and Backend private endpoints **point to the same subresource (**databricks_ui_api), and, since your VNets are peered, you shouldn't have two different ones. You should have one that serves the purpose of providing connectivity to both external users (transit vnet) and Databricks cluster VMs (spoke vnet). Since you already deployed the keyvault private endpoint in a subnet and it fits your purpose, placing it in the same subnet will work, but to make the decision in which subnet to place it, there are several more implications to take into account first.

Browser authentication private endpoint is normally located in the separate transit VNet, since it is only browser/BI Tools/REST Clients that need access to it. Databricks clusters VMs do not need it, so it makes no sense to be in other vnet that is not the transit one.

You should only have one private DNS Zone, but linked to both VNets, since both users and Databricks cluster VMs need to resolve the dns names to the private ips.

Hope this helps. Do let us know if you any further queries.
PRADEEPCHEEKATLA 90,611 Reputation points

2023-05-17T03:53:12.0866667+00:00

@Sunilprasath Elangovan - Following up to see if the above answer was helpful. If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.
Sunilprasath Elangovan 60 Reputation points

2023-07-12T08:41:58.4866667+00:00

Thanks for your answers @PRADEEPCHEEKATLA .

Apologize for responding late.

Yes finally we were able to create the Azure databricks with Private link enabled in Standard deployment.

Your inputs helped us.

And also this Terraform document support us in completing the task.

https://registry.terraform.io/providers/databricks/databricks/latest/docs/guides/azure-private-link-workspace-standard

Regards,

Sunil
PRADEEPCHEEKATLA 90,611 Reputation points

2023-07-13T04:09:22+00:00

@Sunilprasath Elangovan - Glad to know that your issue has been resolved. And thanks for sharing the solution, which might be beneficial to other community members reading this thread.

Share via

How to setup Azure databricks with pubic access disabled and configuring the private DNS for SSO login?

0 additional answers

Your answer