How to install maven library while running databricks notebook using new job cluster through azure data factory?

Question

How to install maven library while running databricks notebook using new job cluster through azure data factory?

Prakamya Aishwarya 121

We are migrating out jobs from interactive cluster to job clusters. I wanted to check if we can specify which libraries to install on a linked service level.

Please note I am already aware of the option to install library from the notebook activity, but do not want to use that since it will require updating lot of activities.

PRADEEPCHEEKATLA 90,641 Reputation points Moderator

2023-05-31T05:01:13.2366667+00:00

@Prakamya Aishwarya - We are reaching out to the internal team to get more information related to your query and will get back to you as soon as we have an update.

2 answers

Your answer

PRADEEPCHEEKATLA 90,641 Reputation points Moderator

2023-05-31T05:01:13.2366667+00:00

@Prakamya Aishwarya - We are reaching out to the internal team to get more information related to your query and will get back to you as soon as we have an update.

Answer 1

VasimTamboli 5,215

To install a Maven library while running a Databricks notebook using a new job cluster through Azure Data Factory, you can follow these steps:

Open your Azure Data Factory instance and go to the "Author & Monitor" section.

Navigate to the pipeline where you want to configure the library installation.

In the pipeline, locate the Databricks notebook activity that you want to run.

Within the Databricks notebook activity, expand the "Advanced" settings.

In the "Advanced" settings, locate the "Libraries" section.

Within the "Libraries" section, click on the "Linked service" drop-down menu and select the linked service associated with your Databricks workspace.

Once the linked service is selected, you will see the option to specify the libraries to be installed. Click on the "Add library" button.

In the library configuration dialog, choose the "Maven" option.

Specify the Maven coordinates for the library you want to install, including the group ID, artifact ID, version, and any other relevant details.

Click on the "OK" button to save the library configuration.

Save and publish the changes to the pipeline.

By specifying the library installation at the linked service level, the library will be automatically installed whenever the Databricks notebook activity runs using the job cluster. This way, you won't need to update individual activities, making it easier to manage library dependencies across multiple pipelines and notebooks.

Prakamya Aishwarya 121 Reputation points

2023-05-27T06:31:05.6666667+00:00

As mentioned in the question description, I am aware of this, but this is not a scalable approach, since it will require going to each and every activity and doing this change.
VasimTamboli 5,215 Reputation points

2023-06-05T05:43:12.7966667+00:00

Create a custom code activity in your Azure Data Factory pipeline. This activity can be a Databricks notebook or a custom code snippet using a supported language like Python or PowerShell.

In the custom code activity, use the Databricks Libraries REST API to programmatically install the Maven libraries. You can make a POST request to the libraries/install endpoint with the Maven coordinates of the libraries you want to install.

You can use the Azure Data Factory pipeline parameters to pass the Maven coordinates as input to the custom code activity, making it dynamic and reusable across different pipelines.

Once the libraries are installed, you can proceed with the remaining steps in your pipeline or execute the Databricks notebook as needed.

By using the Databricks Libraries REST API in a custom code activity, you can automate the library installation process and avoid the need to update individual activities. This approach allows for scalability and flexibility in managing library dependencies across different pipelines and notebooks.

Answer 2

PRADEEPCHEEKATLA 90,641 Moderator

@Prakamya Aishwarya - Thanks for the question and using MS Q&A platform.

You can install a cluster library directly from a public repository such as PyPI or Maven, using a previously installed workspace library, or using an init script.

You can pass the same path of the init scripts in the ADF linked service as shown below.

User's image

When I ran the Databricks Notebook activity, I'm able to install the initscript succesfully without any issue.

User's image

Here is the Event logs of the cluster:

User's image

For more details, refer to Azure Databricks - Cluster libraries.

Hope this helps. Do let us know if you any further queries.

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.

AnnuKumari-MSFT 34,556 Reputation points Microsoft Employee Moderator

2023-06-06T07:40:01.6433333+00:00

Hi Prakamya Aishwarya ,

Just checking in to see if the above answer helped. Please do consider clicking Accept Answer as accepted answers help community as well. Also, please click on Yes for the survey 'Was the answer helpful'
Prakamya Aishwarya 121 Reputation points

2023-06-11T17:49:24.5066667+00:00

Can you please provide me the init script to install maven repo com.microsoft.azure:spark-mssql-connector_2.12_3.0:1.0.0-alpha ? I tried different things but it does not seem to work.

Share via

How to install maven library while running databricks notebook using new job cluster through azure data factory?

2 answers

Your answer