How to install com.crealytics.spark.excel to Synapse Spark Pool

Tamashevich, Tatsiana 60 Reputation points
2024-06-26T14:34:57.57+00:00

Hello,

I have spark pool with following properties:
User's image

According to the instruction mentioned under the Q&A https://learn.microsoft.com/en-us/answers/questions/1185314/how-to-write-data-from-notebook-to-dedicated-sql-p?cid=kerryherger I have installed package com.crealytics.spark.excel that corresponds to spark version:

spark-excel_2.13-3.3.4_0.20.4.

When running the code in the spark Notebook getting an error:

Spark_User_UserApp_NoSuchMethod: Livy session has failed. 

Session state: Dead. 

Error code: Spark_User_UserApp_NoSuchMethod. 

Job failed during run time with state=[dead]. 

 TSG:The application is calling a method that is not available in the Java classpath. This usually means that the application was submitted without a necessary dependency JAR. 

 Likely mitigation steps: 

1. Locate the dependency in question after "java.lang.NoSuchMethodError:" in the driver error stacktrace. 

2. If the dependency is missing, please make sure to include it in your application. 

3. If the dependency is incompatible with Spark and Scala version of your spark pool, please replace the dependency with the compatible versions. 

For example, Spark 2.4 requires Scala 2.11 and Spark 3.1 requires Scala 2.12. Source: User.

Could you, please, clarify what can be wrong and what dependency could be missing?

Thank you very much in advance!

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
4,650 questions
0 comments No comments
{count} votes

Accepted answer
  1. PRADEEPCHEEKATLA-MSFT 84,936 Reputation points Microsoft Employee
    2024-06-27T04:44:29.8366667+00:00

    @Tamashevich, Tatsiana - Thanks for the question and using MS Q&A platform.

    You may checkout my Stackoverflow knowledge articles, which helps to How to read an excel data from ADLS gen2 and write Synapse Dedicated SQL Pool.

    And also, checkout the MS Q&A thread addressing similar issue: Azure Synapse Workspace - How to read an Excel file from Data Lake Gen2 using Pandas or PySpark?.

    Hope this helps. Do let us know if you any further queries.


    If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.


0 additional answers

Sort by: Most helpful