Hello, we have also found this example of using Databricks as a Compute Target for an Azure Machine Learning Pipeline . However, we want to use an existing Databricks Cluster as compute target within Azure Machine Learning Studio for our Azure Machine Learning Pipeline. Could you help us in accomplishing this, please? With best regards Alex

@AlexanderPakakis-0994 Are you looking at adding the cluster from the UI of ML studio rather than using the SDK as mentioned in the notebook you referenced? If Yes, you need to add the same attached compute. Once you select Azure Databricks the following option to add the existing databricks workspace is seen. I hope this helps!! If an answer is helpful, please click on or upvote which might help other community members reading this thread.

Connecting to an existing Databricks Cluster in AMLS

Accepted answer

romungi-MSFT 41,961 Reputation points Microsoft Employee

2022-06-08T05:07:31.167+00:00

@AlexanderPakakis-0994 Are you looking at adding the cluster from the UI of ML studio rather than using the SDK as mentioned in the notebook you referenced?
If Yes, you need to add the same attached compute.

Once you select Azure Databricks the following option to add the existing databricks workspace is seen.

I hope this helps!!

If an answer is helpful, please click on or upvote which might help other community members reading this thread.
Please sign in to rate this answer.
Alexander Pakakis 46 Reputation points

2022-06-08T10:09:07.37+00:00

@romungi-MSFT

I have this already existing Databricks Cluster within Databricks:

I want to add this already existing Databricks Cluster in a Databricks step:

I am aware of the existing Cluster ID (blue rectangle), but I do not know how I define the 'compute_target' (orange rectangle) since the 'compute_target' is the already existing Databricks Cluster (see the first picture).

How can I define the 'compute_target' so that the Databricks step will use the already existing Databricks Cluster?
Could you make create an exemplary code example for this purpose, please?

romungi-MSFT 41,961 Reputation points Microsoft Employee

2022-06-08T12:14:54.153+00:00

@AlexanderPakakis-0994 The compute target is basically the attached compute name that you would like Azure ML workspace to use and link your jobs when you submit these steps.
That is, the compute target can be a new name or your existing data bricks attached compute name in Azure ML that is seen under the compute tab in Azure ML studio.

The cluster id is the one that defines under which cluster of your databricks workspace the job needs to run. I have used the following cluster id from my databricks workspace and defined the compute name as one of the above.

The experiment runs are listed under jobs tab of my compute i used in the above step.

The corresponding runs of databricks are under the Spark UI jobs tab under the cluster that is used in databricks workspace.

Alexander Pakakis 46 Reputation points

2022-06-08T14:02:55.467+00:00

@romungi-MSFT

This solved my issue! Thank you so much!

Alexander Pakakis 46 Reputation points

2022-06-20T13:44:40.207+00:00

@romungi-MSFT

Hello,

attaching a Databricks compute over the AMLS UI is not working anymore:

Could you advise us what we could do, please?
This approach worked until last Wednesday.
Please note: the Databricks Token is valid.

With best regards
Alex

romungi-MSFT 41,961 Reputation points Microsoft Employee

2022-06-20T15:05:28.78+00:00

Tried to attach new compute in the same subscription where Azure ML and Azure Databricks workspace are in one subscription and it worked fine.

If you are using a different subscription then the documentation instructs to grant contributor role on the databricks workspace for the user account.

To attach an Azure Databricks workspace from a different Azure subscription, you (your Azure AD account) must be granted the Contributor role on the Azure Databricks workspace. Check your access in the Azure portal.

Is this the case with your scenario?
If not, I would request to report the issue through Azure support to check what is wrong with your subscription while performing this action.

Alexander Pakakis 46 Reputation points

2022-06-20T15:13:31.157+00:00

@romungi-MSFT

Thank you for your response!
The Databricks and the AMLS workspace are in the same resource group and therefore use the subscription.
I just tried again and it did not work. Maybe it is a regional issue...
However, it worked last week.

I have asked a colleague to try to attach a Databricks compute and he has received the same error as I have.

Alexander Pakakis 46 Reputation points

2022-06-20T15:38:49.913+00:00

@romungi-MSFT

I created a new resource group in the region "East US" and it worked:

I think that it is a regional problem.
Please try out the same steps in North or West Europe.

romungi-MSFT 41,961 Reputation points Microsoft Employee

2022-06-21T09:35:01.79+00:00

Thanks @Alexander Pakakis I think this is a regional issue as you have observed. I am reaching out internally to check what could be wrong while adding workspaces from these regions.

romungi-MSFT 41,961 Reputation points Microsoft Employee

2022-06-23T15:26:48.143+00:00

@Alexander Pakakis The above issue should now be fixed as confirmed by the service team. Could you please check by adding your databricks workspace from effected regions?

Alexander Pakakis 46 Reputation points

2022-06-23T15:28:25.327+00:00

@romungi-MSFT

I can confirm that the issue has been fixed!

Thank you!
Sign in to comment

Connecting to an existing Databricks Cluster in AMLS

0 additional answers