How to install com.crealytics.spark.excel to Synapse Spark Pool

Question

Hello,

I have spark pool with following properties:

According to the instruction mentioned under the Q&A https://learn.microsoft.com/en-us/answers/questions/1185314/how-to-write-data-from-notebook-to-dedicated-sql-p?cid=kerryherger I have installed package com.crealytics.spark.excel that corresponds to spark version:

spark-excel_2.13-3.3.4_0.20.4.

When running the code in the spark Notebook getting an error:

Spark_User_UserApp_NoSuchMethod: Livy session has failed. 

Session state: Dead. 

Error code: Spark_User_UserApp_NoSuchMethod. 

Job failed during run time with state=[dead]. 

 TSG:The application is calling a method that is not available in the Java classpath. This usually means that the application was submitted without a necessary dependency JAR. 

 Likely mitigation steps: 

1. Locate the dependency in question after "java.lang.NoSuchMethodError:" in the driver error stacktrace. 

2. If the dependency is missing, please make sure to include it in your application. 

3. If the dependency is incompatible with Spark and Scala version of your spark pool, please replace the dependency with the compatible versions. 

For example, Spark 2.4 requires Scala 2.11 and Spark 3.1 requires Scala 2.12. Source: User.

Could you, please, clarify what can be wrong and what dependency could be missing?

Thank you very much in advance!

Answer 1

@Tamashevich, Tatsiana - Thanks for the question and using MS Q&A platform.

You may checkout my Stackoverflow knowledge articles, which helps to How to read an excel data from ADLS gen2 and write Synapse Dedicated SQL Pool.

And also, checkout the MS Q&A thread addressing similar issue: Azure Synapse Workspace - How to read an Excel file from Data Lake Gen2 using Pandas or PySpark?.

Hope this helps. Do let us know if you any further queries.

---

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.