Unable to connect to private ADB cluster from ADB connector with Azure IR

Question

Unable to connect to private ADB cluster from ADB connector with Azure IR

Eddie Schneider 20 Microsoft Employee

I'm trying to create a new linked service in Azure Data Factory with ADB connector, and unable to connect to private ADB cluster from ADB connector with Azure IR - I'm getting "Loading failed" message, however with SHIR it works.

Chandra Boorla 14,665 Reputation points Microsoft External Staff Moderator

2025-03-11T23:09:28.6066667+00:00

@Eddie Schneider

The issue you are facing occurs because the Azure Integration Runtime (Azure IR) operates over the public network and cannot connect to resources in a private network (like your Azure Databricks cluster if it’s using a private endpoint).

However, the Self-hosted Integration Runtime (SHIR) works because it is installed within your network (on a VM or on-premises), allowing it to access your private resources.

To resolve this issue, you have a few options

Use a Managed Virtual Network (Managed VNet) - You can configure Azure IR to use a Managed VNet to securely connect to your private Databricks cluster via a Private Endpoint. This setup allows the Azure IR to communicate with your private resources without needing public internet access.

Continue Using SHIR - If using SHIR works for your case and meets your requirements, you can continue using it to connect to the private ADB cluster. SHIR will operate within your network and have the necessary access to your private resources.

Check Private Link Configuration - If you're using Private Link for secure communication, make sure that your private endpoint is set up correctly. You’ll also need to verify that your DNS settings are configured properly to resolve the private IP address of the ADB cluster.

Verify Permissions - Ensure that your Azure IR has the appropriate network permissions to access the private ADB cluster. This includes checking any role assignments and ensuring that the necessary Network Security Group (NSG) rules are in place.

I hope this information helps. Please do let us know if you have any further queries.

Kindly consider upvoting the comment if the information provided is helpful. This can assist other community members in resolving similar issues.

Thank you.
Chandra Boorla 14,665 Reputation points Microsoft External Staff Moderator

2025-03-12T19:23:42.5366667+00:00

@Eddie Schneider

We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution, please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
Chandra Boorla 14,665 Reputation points Microsoft External Staff Moderator

2025-03-13T20:56:08.85+00:00

@Eddie Schneider

We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
Eddie Schneider 20 Reputation points Microsoft Employee

2025-03-17T07:05:27.1966667+00:00

Hi, sorry for the late response.
I already used Managed Vnet and deployed a managed private endpoint in the ADF for the ADB, however the issue was that in the ADB linked service configuration, the list of ADB cluster ids was not presented. The cluster ids were presented only for the SHIR configuration.
The resolution was to manually enter the cluster id. Please validate, we may need some documentation update or notify PG.
Chandra Boorla 14,665 Reputation points Microsoft External Staff Moderator

2025-03-17T19:01:30.0133333+00:00

@Eddie Schneider

I'm glad that you were able to resolve your issue and thank you for posting your solution so that others experiencing the same thing can easily reference this! Since the Microsoft Q&A community has a policy that "The question author cannot accept their own answer. They can only accept answers by others ", I'll repost your solution in case you'd like to accept the answer.

Accepted answer

0 additional answers

Your answer

Chandra Boorla 14,665 Reputation points Microsoft External Staff Moderator

2025-03-11T23:09:28.6066667+00:00

@Eddie Schneider

The issue you are facing occurs because the Azure Integration Runtime (Azure IR) operates over the public network and cannot connect to resources in a private network (like your Azure Databricks cluster if it’s using a private endpoint).

However, the Self-hosted Integration Runtime (SHIR) works because it is installed within your network (on a VM or on-premises), allowing it to access your private resources.

To resolve this issue, you have a few options

Use a Managed Virtual Network (Managed VNet) - You can configure Azure IR to use a Managed VNet to securely connect to your private Databricks cluster via a Private Endpoint. This setup allows the Azure IR to communicate with your private resources without needing public internet access.

Continue Using SHIR - If using SHIR works for your case and meets your requirements, you can continue using it to connect to the private ADB cluster. SHIR will operate within your network and have the necessary access to your private resources.

Check Private Link Configuration - If you're using Private Link for secure communication, make sure that your private endpoint is set up correctly. You’ll also need to verify that your DNS settings are configured properly to resolve the private IP address of the ADB cluster.

Verify Permissions - Ensure that your Azure IR has the appropriate network permissions to access the private ADB cluster. This includes checking any role assignments and ensuring that the necessary Network Security Group (NSG) rules are in place.

I hope this information helps. Please do let us know if you have any further queries.

Kindly consider upvoting the comment if the information provided is helpful. This can assist other community members in resolving similar issues.

Thank you.
Chandra Boorla 14,665 Reputation points Microsoft External Staff Moderator

2025-03-12T19:23:42.5366667+00:00

@Eddie Schneider

We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution, please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
Chandra Boorla 14,665 Reputation points Microsoft External Staff Moderator

2025-03-13T20:56:08.85+00:00

@Eddie Schneider

We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
Eddie Schneider 20 Reputation points Microsoft Employee

2025-03-17T07:05:27.1966667+00:00

Hi, sorry for the late response.
I already used Managed Vnet and deployed a managed private endpoint in the ADF for the ADB, however the issue was that in the ADB linked service configuration, the list of ADB cluster ids was not presented. The cluster ids were presented only for the SHIR configuration.
The resolution was to manually enter the cluster id. Please validate, we may need some documentation update or notify PG.
Chandra Boorla 14,665 Reputation points Microsoft External Staff Moderator

2025-03-17T19:01:30.0133333+00:00

@Eddie Schneider

I'm glad that you were able to resolve your issue and thank you for posting your solution so that others experiencing the same thing can easily reference this! Since the Microsoft Q&A community has a policy that "The question author cannot accept their own answer. They can only accept answers by others ", I'll repost your solution in case you'd like to accept the answer.

Answer 1

@Eddie Schneider

I'm glad that you were able to resolve your issue and thank you for posting your solution so that others experiencing the same thing can easily reference this! Since the Microsoft Q&A community has a policy that "The question author cannot accept their own answer. They can only accept answers by others ", I'll repost your solution in case you'd like to accept the answer.

Issue:

Unable to connect to private ADB cluster from ADB connector with Azure IR

Solution:

"Instead of relying on the UI to fetch the cluster IDs, the user manually entered the cluster ID into the Linked Service configuration, which resolved the issue."

If I missed anything please let me know and I'd be happy to add it to my answer, or feel free to comment below with any additional information.

Feedback

Appreciate if you could share the feedback on our feedback channel. Which would be open for the user community to upvote & comment on. This allows our product teams to effectively prioritize your request against our existing feature backlog and gives insight into the potential impact of implementing the suggested feature.

Hope this helps. Do let us know if you have any further queries.

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.

Share via

Unable to connect to private ADB cluster from ADB connector with Azure IR

0 additional answers

Your answer