ErrorCode=FailedToConnectToDatabricksWorkspace,Failed to connect to Databricks workspace. Error Cluster 19628023f7cbfc49 does not exist. Cluster 19628023f7cbfc49 does not exist Processed HTTP request failed

Question

Hi,

I'm trying to connect to a databricks catalog from Fabric, and cannot connect via Pipelines. It does work for dataflow, but pipeline always throws me this error from the same connection.

Answer 1

Before I jump into researching, how come I can connect to the databricks workspace using a Gen2 Dataflow in Fabric, but cannot connect to it via a Fabric Pipeline?

Answer 2

Hi Ryan White,

It sounds like you're running into an issue where your connection to the Databricks workspace is failing within Azure Data Factory Pipelines, specifically with the error indicating that the cluster does not exist. Here are a few steps you can try to troubleshoot this issue:

1. Verify Cluster ID: Make sure that the cluster ID 19628023f7cbfc49 you are trying to connect to is correct and currently active. You can check the status of your clusters in the Azure Databricks workspace.
2. Check IP Access List:
   • Ensure that the IP Access List feature is enabled in your Databricks workspace. If it is enabled, you need to allowlist the Azure Data Factory IP ranges to permit traffic. You can find detailed instructions on this here: Check if IP Access List feature is enabled.
   • If necessary, update the IP Access Lists to include the relevant Azure Data Factory IP ranges for your region. You can find the correct IP ranges by downloading the latest Azure IP Ranges and Service Tags document here.
3. User-Defined Routes (UDRs): It’s recommended to add a user-defined route (UDR) in Azure pointing to the AzureDatabricks service tag for better connectivity. This ensures your workspace can connect to the required resources. More details can be found in the documentation: Configure user-defined routes with Azure service tags.
4. Check Network Configuration: Ensure there are no firewall rules blocking access and that any custom DNS configurations allow for resolution of necessary services. For example, your DNS setup should allow access to Azure-specific hosts required for Databricks.
5. Other Recommendations: If you've recently made changes to your Databricks configurations or if your workspace is behind a VNet, consider checking cluster dependencies and ensuring that all necessary DNS and access configurations are correctly set up.

If you've gone through these steps and the problem persists, some follow-up questions might help narrow it down:

• Are you able to connect to any other clusters or is it just this specific cluster?
• Have there been any recent changes to your network settings or Databricks configuration?
• Can you check the cluster status in the Azure Databricks workspace to confirm it's up and running?

Hope this helps you get to the bottom of the issue! Let me know if you have any further questions.

References:

• Troubleshoot Databricks workspace connectivity issues with Azure Data Factory
• Check if IP Access List feature is enabled
• Azure IP Ranges and Service Tags – Public Cloud
• Configure user-defined routes with Azure service tags