How to use AWS Databricks note book in ADF..?

Naveen Kumar Rachapalle (EXT/DIGITAL) 25

Hi Team,

I want to create Linked service in ADF with AWS Databricks notebook. Can you guys help me with list of steps that i need follow.

Regards,

Naveen.

PRADEEPCHEEKATLA-MSFT 77,751 Reputation points Microsoft Employee

2023-10-30T09:00:32.85+00:00

@Naveen Kumar Rachapalle (EXT/DIGITAL) - Thanks for the question and using MS Q&A platform.

Since, I don't have AWS environment to test this scenario. But I can provide you the overview on how to use AWS Databricks notebook in Azure Data Factory.

Before we proceed, may I know if you have already created an AWS Databricks workspace and a notebook in it?

Important!: While creating a linked service in ADF try to use account selection method:manual and enter the

AWS Databricks workspace url and authentication type Access token (Use PAT token)

Here are the steps to create a Linked Service in ADF with AWS Databricks notebook:

1.     Go to the Manage hub in your ADF instance and click on the Linked Services tab under connections.

2.     Click on the New button and select Azure Databricks from the list of available connectors.

3.     In the New Linked Service (Azure Databricks) window, provide a name for your Linked Service and select the appropriate workspace type (Azure or AWS).

4.     In the Workspace URL field, enter the URL of your AWS Databricks workspace.

5.     In the Access Token field, enter the access token for your AWS Databricks workspace.

6.     In the Select cluster field, select as per your requirement.

7.     Click on the Test Connection button to verify that the connection is successful.

8.     Once the connection is verified, click on the Create button to create the Linked Service.

9.     You can now use this Linked Service in your ADF pipelines to run your AWS Databricks notebook.

Once the linked service is created, you can create a new pipeline and select Notebook under Databricks activity.

Under Azure Databricks: Select the databricks linked service created.

Under Settings: Select the notebook path

For more details, refer to Run a Databricks notebook with the Databricks Notebook Activity in Azure Data Factory.

Hope this helps. Do let us know if you any further queries.
Naveen Kumar Rachapalle (EXT/DIGITAL) 25 Reputation points

2023-10-30T09:51:09.6466667+00:00

Hi Team

I have Tried Same steps as per Suggestion and its Not Working.

Below is the Screenshot.
PRADEEPCHEEKATLA-MSFT 77,751 Reputation points Microsoft Employee

2023-10-30T11:26:05.6666667+00:00

@Naveen Kumar Rachapalle (EXT/DIGITAL) - Thanks for the update.

We are reaching out to the internal team to get more information related to your query and will get back to you as soon as we have an update.
Naveen Kumar Rachapalle (EXT/DIGITAL) 25 Reputation points

2023-10-31T07:55:31.32+00:00

@PRADEEPCHEEKATLA-MSFT any update..?
PRADEEPCHEEKATLA-MSFT 77,751 Reputation points Microsoft Employee

2023-11-06T06:53:15.8833333+00:00

@Naveen Kumar Rachapalle (EXT/DIGITAL) - Just checking in to see if the below answer helped. If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.

Accepted answer

PRADEEPCHEEKATLA-MSFT 77,751 Reputation points Microsoft Employee

2023-11-03T05:27:50.5166667+00:00

@Naveen Kumar Rachapalle (EXT/DIGITAL) - Thanks for the question and using MS Q&A platform.

Here is the confirmation from the internal team:

This is an unsupported scenario by design for security concerns which disallows AWS or any other cloud Databricks connection via ADF linked service. It is required to use Azure Databricks to connect to ADF.

Appreciate if you could share the feedback on our feedback channel. Which would be open for the user community to upvote & comment on. This allows our product teams to effectively prioritize your request against our existing feature backlog and gives insight into the potential impact of implementing the suggested feature.

Hope this helps. Do let us know if you any further queries.

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.
Please sign in to rate this answer.
PRADEEPCHEEKATLA-MSFT 77,751 Reputation points Microsoft Employee

2023-11-10T08:26:47.4+00:00

@Naveen Kumar Rachapalle (EXT/DIGITAL) - Following up to see if the above answer was helpful. If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.

Syed Ahamed S 0 Reputation points

2024-04-25T10:09:37.6533333+00:00

Any other possible approaches to achieve the same?
Sign in to comment

How to use AWS Databricks note book in ADF..?

0 additional answers