Cannot install Elasticsearch library from Maven

Question

I have a databricks cluster for which I want to install the Elasticsearch library from Maven via the UI but for which I receive an error. Installing libraries by uploading jars works fine and via Pypi also works.

Cluster details:

• Single node type DS12_v2
• 10.4 LTS (includes Apache Spark 3.2.1, Scala 2.12)
• Spark config:
  spark.master local[*]
  spark.databricks.passthrough.enabled false
  spark.databricks.cluster.profile singleNode
  spark.databricks.delta.preview.enabled true

Steps followed:

1. In the databricks UI, go to clusters and select the correct cluster
2. Go to libraries>install new>Maven>Search Packages
3. Search on elasticsearch, select the library with coordinates: "org.elasticsearch:elasticsearch:8.2.0"
4. Click install

Error message:
Library installation attempted on the driver node of cluster 0616-092931-8wtrlmkm and failed. Please refer to the following error message to fix the library or contact Databricks support. Error Code: DRIVER_LIBRARY_INSTALLATION_FAILURE. Error Message: Library resolution failed. Cause: java.lang.RuntimeException: org.apache.lucene:lucene-core download failed.
at com.databricks.libraries.server.MavenInstaller.$anonfun$resolveDependencyPaths$5(MavenLibraryResolver.scala:275)
at scala.collection.immutable.HashMap$HashTrieMap.getOrElse0(HashMap.scala:596)
at scala.collection.immutable.HashMap.getOrElse(HashMap.scala:73)
at com.databricks.libraries.server.MavenInstaller.$anonfun$resolveDependencyPaths$4(MavenLibraryResolver.scala:275)
at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:286)
at scala.collection.mutable.ArraySeq.foreach(ArraySeq.scala:75)
at scala.collection.TraversableLike.map(TraversableLike.scala:286)
at scala.collection.TraversableLike.map$(TraversableLike.scala:279)
at scala.collection.AbstractTraversable.map(Traversable.scala:108)
at com.databricks.libraries.server.MavenInstaller.resolveDependencyPaths(MavenLibraryResolver.scala:271)
at com.databricks.libraries.server.MavenInstaller.doDownloadMavenPackages(MavenLibraryResolver.scala:481)
at com.databricks.libraries.server.MavenInstaller.$anonfun$downloadMavenPackages$3(MavenLibraryResolver.scala:400)
at com.databricks.backend.common.util.FileUtils$.withTemporaryDirectory(FileUtils.scala:468)
at com.databricks.libraries.server.MavenInstaller.$anonfun$downloadMavenPackages$2(MavenLibraryResolver.scala:399)
at com.databricks.logging.UsageLogging.$anonfun$recordOperation$1(UsageLogging.scala:413)
at com.databricks.logging.UsageLogging.executeThunkAndCaptureResultTags$1(UsageLogging.scala:507)
at com.databricks.logging.UsageLogging.$anonfun$recordOperationWithResultTags$4(UsageLogging.scala:528)
at com.databricks.logging.Log4jUsageLoggingShim$.$anonfun$withAttributionContext$1(Log4jUsageLoggingShim.scala:53)
at scala.util.DynamicVariable.withValue(DynamicVariable.scala:62)
at com.databricks.logging.AttributionContext$.withValue(AttributionContext.scala:94)
at com.databricks.logging.Log4jUsageLoggingShim$.withAttributionContext(Log4jUsageLoggingShim.scala:49)
at com.databricks.logging.UsageLogging.withAttributionContext(UsageLogging.scala:283)
at com.databricks.logging.UsageLogging.withAttributionContext$(UsageLogging.scala:282)
at com.databricks.libraries.server.MavenInstaller.withAttributionContext(MavenLibraryResolver.scala:61)
at com.databricks.logging.UsageLogging.withAttributionTags(UsageLogging.scala:318)
at com.databricks.logging.UsageLogging.withAttributionTags$(UsageLogging.scala:303)
at com.databricks.libraries.server.MavenInstaller.withAttributionTags(MavenLibraryResolver.scala:61)
at com.databricks.logging.UsageLogging.recordOperationWithResultTags(UsageLogging.scala:502)
at com.databricks.logging.UsageLogging.recordOperationWithResultTags$(UsageLogging.scala:422)
at com.databricks.libraries.server.MavenInstaller.recordOperationWithResultTags(MavenLibraryResolver.scala:61)
at com.databricks.logging.UsageLogging.recordOperation(UsageLogging.scala:413)
at com.databricks.logging.UsageLogging.recordOperation$(UsageLogging.scala:385)
at com.databricks.libraries.server.MavenInstaller.recordOperation(MavenLibraryResolver.scala:61)
at com.databricks.libraries.server.MavenInstaller.downloadMavenPackages(MavenLibraryResolver.scala:398)
at com.databricks.libraries.server.MavenInstaller.downloadMavenPackagesWithRetry(MavenLibraryResolver.scala:153)
at com.databricks.libraries.server.MavenInstaller.resolveMavenPackages(MavenLibraryResolver.scala:117)
at com.databricks.libraries.server.MavenLibraryResolver.resolve(MavenLibraryResolver.scala:48)
at com.databricks.libraries.server.ManagedLibraryManager$GenericManagedLibraryResolver.resolve(ManagedLibraryManager.scala:252)
at com.databricks.libraries.server.ManagedLibraryManagerImpl$.$anonfun$resolvePrimitives$1(ManagedLibraryManagerImpl.scala:1229)
at com.databricks.libraries.server.ManagedLibraryManagerImpl$.$anonfun$resolvePrimitives$1$adapted(ManagedLibraryManagerImpl.scala:1224)
at scala.collection.immutable.List.foreach(List.scala:431)
at com.databricks.libraries.server.ManagedLibraryManagerImpl$.resolvePrimitives(ManagedLibraryManagerImpl.scala:1224)
at com.databricks.clusterlibraries.memorystate.PerClusterInMemoryMavenLibraryState.resolveMavenLibraries(PerClusterInMemoryMavenLibraryState.scala:340)
at com.databricks.clusterlibraries.memorystate.PerClusterInMemoryMavenLibraryState.resolveAndStoreMavenLibraries(PerClusterInMemoryMavenLibraryState.scala:199)
at com.databricks.clusterlibraries.memorystate.PerClusterInMemoryMavenLibraryState.$anonfun$getJarLibrariesToInstallAndMavenLibrariesToUpdate$3(PerClusterInMemoryMavenLibraryState.scala:122)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
at scala.concurrent.Future$.$anonfun$apply$1(Future.scala:659)
at scala.util.Success.$anonfun$map$1(Try.scala:255)
at scala.util.Success.map(Try.scala:213)
at scala.concurrent.Future.$anonfun$map$1(Future.scala:292)
at scala.concurrent.impl.Promise.liftedTree1$1(Promise.scala:33)
at scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:33)
at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:64)
at com.databricks.threading.NamedExecutor$$anon$2.$anonfun$run$1(NamedExecutor.scala:360)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
at com.databricks.logging.Log4jUsageLoggingShim$.$anonfun$withAttributionContext$1(Log4jUsageLoggingShim.scala:53)
at scala.util.DynamicVariable.withValue(DynamicVariable.scala:62)
at com.databricks.logging.AttributionContext$.withValue(AttributionContext.scala:94)
at com.databricks.logging.Log4jUsageLoggingShim$.withAttributionContext(Log4jUsageLoggingShim.scala:49)
at com.databricks.logging.UsageLogging.withAttributionContext(UsageLogging.scala:283)
at com.databricks.logging.UsageLogging.withAttributionContext$(UsageLogging.scala:282)
at com.databricks.threading.NamedExecutor.withAttributionContext(NamedExecutor.scala:287)
at com.databricks.threading.NamedExecutor$$anon$2.run(NamedExecutor.scala:359)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)

Answer 1

Hello @Verret, Lukas ,

Thanks for the question and using MS Q&A platform.

As per the repro from my end, I'm unable to install any version of elasticsearch from org.elasticsearch. Seems like there is a problem with a particular maven jar (org.elasticsearch).

How to install org.elasticsearch:elasticsearch:8.2.0 on databricks cluster?

You can manually download the jar from the maven repository: https://mvnrepository.com/artifact/org.elasticsearch/elasticsearch and the upload to the cluster.

Go to Libraries => Install new => Drop JAR here => select the downloaded jar => click install

Hope this will help. Please let us know if any further queries.

------------------------------

• Please don't forget to click on or upvote button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
• Want a reminder to come back and check responses? Here is how to subscribe to a notification
• If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators