Azure Databricks fail to install Geospark libraries from Maven

Anuj, Singh (Cognizant) 5

Hi Team , I am attempting to add below two geospark Maven libraries to my Azure Databricks interactive cluster with Runtime Version 14.3 LTS . Geospark_Library

However , I am getting below error

Library installation attempted on the driver node of cluster 0311-204237-y518gt4u and failed.

Please refer to the following error message to fix the library or contact Databricks support.

Error Code: DRIVER_LIBRARY_INSTALLATION_FAILURE.

Error Message: Library resolution failed because unresolved dependency: org.datasyslab:geospark-sql_2.3:1.3.1: not found unresolved dependency: org.datasyslab:geospark:1.3.1: not found

PRADEEPCHEEKATLA-MSFT 77,516 Reputation points Microsoft Employee

2024-04-18T09:05:34.1766667+00:00

@Anuj, Singh (Cognizant) - Just checking in to see if the below answer helped. If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.

1 answer

PRADEEPCHEEKATLA-MSFT 77,516 Reputation points Microsoft Employee

2024-04-15T07:06:28.4366667+00:00
@Anuj, Singh (Cognizant) - Thanks for the question and using MS Q&A platform.

The error message you are seeing indicates that Databricks is unable to download the Geospark library from Maven. This could be due to a network issue or a problem with the Maven repository.

Here are some steps you can take to resolve this issue:

Check your network connectivity: Ensure that your network connection is stable and that you are able to access the internet. You can try pinging the Maven repository to see if you are able to connect to it.

Check the Maven repository: Verify that the Maven repository is up and running and that the GeoSpark library is available. You can try downloading the library manually from the repository to see if it is accessible.

Check the Databricks cluster configuration: Ensure that the cluster is configured to use the correct Maven repository and that the repository is accessible from the cluster. You can check the cluster configuration by going to the cluster settings in the Databricks workspace.

Try using a different Maven repository: If the issue persists, you can try using a different Maven repository to see if that resolves the issue. You can configure the cluster to use a different repository by updating the Maven settings in the cluster configuration.

As per the repro, I had tried to install the maven library (org.datasyslab:geospark:1.3.1 & org.datasyslab:geospark-sql_2.3:1.3.1) on Cluster Details - Databricks Runtime Version: 14.3 LTS (includes Apache Spark 3.5.0, Scala 2.12) and able to successfully able to install without any issue.

In case, if you are still experiencing the same issue. I would suggest you share the steps which you are trying to install and also try to download the jar file directly from the maven site: https://mvnrepository.com/artifact/org.datasyslab and upload manally and install.

Hope this helps. Do let us know if you any further queries.

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.
Please sign in to rate this answer.
Anuj, Singh (Cognizant) 5 Reputation points

2024-04-16T06:20:25.29+00:00

Check your network connectivity - Yes, I have checked. No issues with network connectivity.

Check the Maven repository - How to verify that the Maven repository is up and running and that the GeoSpark library is available?

Check the Databricks cluster configuration: Yes, I have checked the Databricks cluster configuration. Since my cluster is Unity Catalog enabled, I have added maven library (org.datasyslab:geospark:1.3.1 & org.datasyslab:geospark-sql_2.3:1.3.1) in Allow List. I realized that the allowlist needs to apply to interactive clusters configured with Unity Catalog in ’shared’ access mode'. Once added I restarted the clusters for changes to take effect.

The issue still persists.

Can you suggest which different Maven repository?

PRADEEPCHEEKATLA-MSFT 77,516 Reputation points Microsoft Employee

2024-04-16T06:42:34.51+00:00

@Anuj, Singh (Cognizant) - In case, if you are still experiencing the same issue. I would suggest you share the steps which you are trying to install and also try to download the jar file directly from the maven site: https://mvnrepository.com/artifact/org.datasyslab and upload manually and install.

Anuj, Singh (Cognizant) 5 Reputation points

2024-04-16T10:07:55.21+00:00

~~Can you share the steps to download the jar file directly from the maven site: https://mvnrepository.com/artifact/org.datasyslab and upload manually and install~~

Anuj, Singh (Cognizant) 5 Reputation points

2024-04-16T10:46:56.7366667+00:00

I have checked the Databricks cluster configuration: My cluster is Unity Catalog enabled; I have added maven library (org.datasyslab:geospark:1.3.1 & org.datasyslab:geospark-sql_2.3:1.3.1) in Allow List. I realized that the allowlist needs to apply to interactive clusters configured with Unity Catalog in ’shared’ access mode'. Once added I restarted the clusters for changes to take effect.

I am encountering issues. Can you suggest how to ensure that the Databricks Runtime Version is compatible with the library version we are trying to install. In my case, Databricks Runtime Version is 14.3 LTS (includes Apache Spark 3.5.0, Scala 2.12) and Library version org.datasyslab:geospark:1.3.1 & org.datasyslab:geospark-sql_2.3:1.3.1

Please note I can consider to downgrading my Databricks Runtime Version to one that includes Apache Spark 2.X and Scala 2.3 . Please suggest what would be the correct Databricks Runtime Version

PRADEEPCHEEKATLA-MSFT 77,516 Reputation points Microsoft Employee

2024-04-17T03:10:21.6033333+00:00

@Anuj, Singh (Cognizant) - There are different methods to install packages in Azure Databricks:

GUI Method

Method1: Using libraries

To make third-party or locally-built code available to notebooks and jobs running on your clusters, you can install a library. Libraries can be written in Python, Java, Scala, and R. You can upload Java, Scala, and Python libraries and point to external packages in PyPI, Maven, and CRAN repositories.

Steps to install third party libraries:

Step1: Create Databricks Cluster.

Step2: Select the cluster created.

Step3: Select Libraries => Install New => Select Library Source = "Maven" => Coordinates => Search Packages => Select Maven Central => Search for the package required. Example: (GDAL) => Select the version (3.0.0) required => Install

Notebook methods

Method2: Using Cluster-scoped init scripts

Cluster-scoped init scripts are init scripts defined in a cluster configuration. Cluster-scoped init scripts apply to both clusters you create and those created to run jobs. Since the scripts are part of the cluster configuration, cluster access control lets you control who can change the scripts.

Step1: Add the DBFS path dbfs:/databricks/scripts/gdal_install.sh to the cluster init scripts

# --- Run 1x to setup the init script. --- # Restart cluster after running. dbutils.fs.put("/databricks/scripts/gdal_install.sh",""" #!/bin/bash sudo add-apt-repository ppa:ubuntugis/ppa sudo apt-get update sudo apt-get install -y cmake gdal-bin libgdal-dev python3-gdal""", True)

Step2: Restart the cluster after running step1 for the first time.

Method3: Python packages are installed in the Spark container using pip install.

Using pip to install "psutil" library.

CLI & API Methods

Method4: Libraries CLI

You run Databricks libraries CLI subcommands by appending them to databricks libraries.

databricks libraries -h

Install a JAR from DBFS:

databricks libraries install --cluster-id $CLUSTER_ID --jar dbfs:/test-dir/test.jar

Method6: Libraries API

The Libraries API allows you to install and uninstall libraries and get the status of libraries on a cluster.

Install libraries on a cluster. The installation is asynchronous - it completes in the background after the request. 2.0/libraries/install

Example Request:

{ "cluster_id": "10201-my-cluster", "libraries": [ { "jar": "dbfs:/mnt/libraries/library.jar" }, { "egg": "dbfs:/mnt/libraries/library.egg" }, { "whl": "dbfs:/mnt/libraries/mlflow-0.0.1.dev0-py2-none-any.whl" }, { "whl": "dbfs:/mnt/libraries/wheel-libraries.wheelhouse.zip" }, { "maven": { "coordinates": "org.jsoup:jsoup:1.7.2", "exclusions": ["slf4j:slf4j"] } }, { "pypi": { "package": "simplejson", "repo": "https://my-pypi-mirror.com" } }, { "cran": { "package": "ada", "repo": "https://cran.us.r-project.org" } } ] }

Hope this helps. Do let us know if you any further queries.

PRADEEPCHEEKATLA-MSFT 77,516 Reputation points Microsoft Employee

2024-04-19T08:19:03.06+00:00

@Anuj, Singh (Cognizant) - We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.

Anuj, Singh (Cognizant) 5 Reputation points

2024-04-22T04:17:58.1766667+00:00

I attempted to create different computes with Databricks Runtime Version 14.1 /14.3 & 15.0 compute and tried to add only org.datasyslab:geospark:1.3.1 library in each of them however the issue persists, I also tried to add only org.datasyslab:geospark:1.3.1 library on our existing unity catalog enabled compute i.e. 14.3 LTS. Once added I restarted the clusters for changes to take effect but the issue still persists

PRADEEPCHEEKATLA-MSFT 77,516 Reputation points Microsoft Employee

2024-04-22T06:54:43.6166667+00:00

@Anuj, Singh (Cognizant) - I agree that this issue looks strange and I wasn't able to reproduce this issue. If you have a support plan could you please file a support ticket for deeper investigation and do share the SR# with us?

Anuj, Singh (Cognizant) 5 Reputation points

2024-04-22T14:09:41.6666667+00:00

@PRADEEPCHEEKATLA-MSFT Raised a Microsoft ticket with #Case 2404150050000403

Anuj, Singh (Cognizant) 5 Reputation points

2024-04-22T14:10:27.0533333+00:00

Raised a Microsoft ticket with #Case 2404150050000403

PRADEEPCHEEKATLA-MSFT 77,516 Reputation points Microsoft Employee

2024-04-23T03:38:08.5466667+00:00

@Anuj, Singh (Cognizant) - Thanks for creating a support ticket. Once the issue has been sorted out with the help of support, please do share the resolution, which might be beneficial to other community members reading this thread.
Sign in to comment