Library installation fails in databricks from Maven due to com.sun.mail:javax.mail

Question

I have been trying to install com.microsoft.azure:azure-eventhubs-spark_2.12:2.3.22 in databricks. It has been failing continuously. It works only few times and then when the cluster is restarted due to some reason, it fails again. This is causing issues in jobs running.

Below is the error message.

Library installation attempted on the driver node of cluster and failed. Please refer to the following error message to fix the library or contact Databricks support. Error Code: DRIVER_LIBRARY_INSTALLATION_FAILURE. Error Message: Library resolution failed because com.sun.mail:javax.mail download failed.

I am trying to check this library from maven - https://mvnrepository.com/artifact/com.sun.mail/javax.mail but it says this has been moved. Is this due to this issue? What is the permanent fix so that it doesnt fail again?

Databricks Run Time Version : 12.2 LTS (includes Apache Spark 3.3.2, Scala 2.12)

Let me kow if any other details are required.

Answer 1

@Dilisha Jain Kala - Thanks for the question and using MS Q&A platform.

The error message you are seeing indicates that Databricks is unable to download the javax.mail library from Maven. This could be due to a network issue or a problem with the Maven repository.

Here are some steps you can take to resolve this issue:

1. Check your network connectivity: Ensure that your network connection is stable and that you are able to access the internet. You can try pinging the Maven repository to see if you are able to connect to it.
2. Check the Maven repository: Verify that the Maven repository is up and running and that the javax.mail library is available. You can try downloading the library manually from the repository to see if it is accessible.
3. Check the Databricks cluster configuration: Ensure that the cluster is configured to use the correct Maven repository and that the repository is accessible from the cluster. You can check the cluster configuration by going to the cluster settings in the Databricks workspace.
4. Try using a different Maven repository: If the issue persists, you can try using a different Maven repository to see if that resolves the issue. You can configure the cluster to use a different repository by updating the Maven settings in the cluster configuration.

As per the repro, I had tried to install the maven library (com.microsoft.azure:azure-eventhubs-spark_2.12:2.3.22) on Cluster Details - Databricks Runtime Version: 12.2 LTS (includes Apache Spark 3.3.2, Scala 2.12) and able to successfully able to install wihtout any issue.

In case, if you are still experiencing the same issue. I would suggest you to share the databricks runtime version which you are using to install the maven library.

Hope this helps. Do let us know if you any further queries.

---

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.