Error writing excel file from Pyspark

Naveen Kuppili 41 Reputation points
2021-12-12T17:48:49.213+00:00

I'm trying to generate excel file from spark dataframe and it throws me with below error. I am not sure where I am getting this wrong, any ideas?

df_spark.write.format("com.crealytics.spark.excel")\
  .option("header", "true")\
  .mode("overwrite")\
  .save(path)

databricks runtime version - 9.1 LTS (includes Apache Spark 3.1.2, Scala 2.12)
libraries - com.crealytics:spark-excel_2.12:3.1.2_0.16.0

error message -

Py4JJavaError: An error occurred while calling o437.save.
: java.lang.NoSuchMethodError: org.apache.commons.io.IOUtils.byteArray(I)[B

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,062 questions
0 comments No comments
{count} votes

Accepted answer
  1. PRADEEPCHEEKATLA-MSFT 85,026 Reputation points Microsoft Employee
    2021-12-13T08:50:39.563+00:00

    Hello @Naveen Kuppili ,

    Thanks for the question and using MS Q&A platform.

    As per the repro from our end, it looks like this is an issue with package com.crealytics.spark.excel.

    157106-image.png

    You may checkout the answer provided on the Github issue: Reading excel file in Azure Databricks.

    In case, if you get the same error after follow the workaround provided by nightscape, I would suggest you to reply on the same issue or create a new issue.

    Hope this will help. Please let us know if any further queries.

    ------------------------------

    • Please don't forget to click on 130616-image.png or upvote 130671-image.png button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
    • Want a reminder to come back and check responses? Here is how to subscribe to a notification
    • If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators
    0 comments No comments

0 additional answers

Sort by: Most helpful