Error writing excel file from Pyspark

Question

I'm trying to generate excel file from spark dataframe and it throws me with below error. I am not sure where I am getting this wrong, any ideas?

df_spark.write.format("com.crealytics.spark.excel")\
  .option("header", "true")\
  .mode("overwrite")\
  .save(path)

databricks runtime version - 9.1 LTS (includes Apache Spark 3.1.2, Scala 2.12)
libraries - com.crealytics:spark-excel_2.12:3.1.2_0.16.0

error message -

Py4JJavaError: An error occurred while calling o437.save.
: java.lang.NoSuchMethodError: org.apache.commons.io.IOUtils.byteArray(I)[B

Answer 1

Hello @Naveen Kuppili ,

Thanks for the question and using MS Q&A platform.

As per the repro from our end, it looks like this is an issue with package com.crealytics.spark.excel.

You may checkout the answer provided on the Github issue: Reading excel file in Azure Databricks.

In case, if you get the same error after follow the workaround provided by nightscape, I would suggest you to reply on the same issue or create a new issue.

Hope this will help. Please let us know if any further queries.

------------------------------

• Please don't forget to click on or upvote button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
• Want a reminder to come back and check responses? Here is how to subscribe to a notification
• If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators