Setting up communication between my application and databricks cluster

Question

Setting up communication between my application and databricks cluster

Karn 5

My application is hosted on azure account 1 and I am trying to launch job in azure databricks cluster - my job needs to communicate back to our hosted env to fetch some metadata and update job metrics.

We are thinking of using azure private link for bi-directional communication b/w my app and DB cluster. Can anyone help in providing the reference architecture or guide if this is a right approach.

KapilAnanth 49,866 Reputation points Moderator

2023-06-06T08:46:51.31+00:00
@Karn

Welcome to the Microsoft Q&A Platform. Thank you for reaching out & I hope you are doing well.

I understand that you would like to establish private connectivity between your application and azure databricks cluster.

From networking perspective,

When you say "bi-directional communication"

Do you mean that the traffic needs to be initiated from both the sides?

If that's the case, then I am afraid this won't be feasible

Please note that any PaaS service's Private EndPoint will create a NIC in a VNet.

Now, resources connected to this VNet (such as VMs and Vnet Integrated services) can use this NIC to access the PaaS service.

However, if you want the PaaS service to initiate the connection and talk to resources deployed in the VNet - this is not doable with Azure Private EndPoint

Hope this helps.

Cheers,

Kapil
KapilAnanth 49,866 Reputation points Moderator

2023-06-09T06:35:01.05+00:00

@Karn

May I know if you got a chance to review my previous comment?

Please let me know if you are facing any challenges or if there are any follow-up questions, I shall be glad to address them.

Thanks,

Kapil
KapilAnanth 49,866 Reputation points Moderator

2023-06-12T04:53:45.2066667+00:00

@Karn

Reaching out to check if there are any questions on this.

Please let us know if we can be of any further assistance here.

Thanks,

Kapil