Hello,
I have spark pool with following properties:
According to the instruction mentioned under the Q&A https://learn.microsoft.com/en-us/answers/questions/1185314/how-to-write-data-from-notebook-to-dedicated-sql-p?cid=kerryherger I have installed package com.crealytics.spark.excel that corresponds to spark version:
spark-excel_2.13-3.3.4_0.20.4.
When running the code in the spark Notebook getting an error:
Spark_User_UserApp_NoSuchMethod: Livy session has failed.
Session state: Dead.
Error code: Spark_User_UserApp_NoSuchMethod.
Job failed during run time with state=[dead].
TSG:The application is calling a method that is not available in the Java classpath. This usually means that the application was submitted without a necessary dependency JAR.
Likely mitigation steps:
1. Locate the dependency in question after "java.lang.NoSuchMethodError:" in the driver error stacktrace.
2. If the dependency is missing, please make sure to include it in your application.
3. If the dependency is incompatible with Spark and Scala version of your spark pool, please replace the dependency with the compatible versions.
For example, Spark 2.4 requires Scala 2.11 and Spark 3.1 requires Scala 2.12. Source: User.
Could you, please, clarify what can be wrong and what dependency could be missing?
Thank you very much in advance!