Share via

Arrow-Optimization in Python Notebook fails

Marek, Phil 21 Reputation points
2022-05-17T10:18:10.933+00:00

Hi,

since a few days I'm getting the following warning when trying to convert from and to pandas in Pyspark Notebooks.

/opt/spark/python/lib/pyspark.zip/pyspark/sql/pandas/conversion.py:329: UserWarning: createDataFrame attempted Arrow optimization because 'spark.sql.execution.arrow.pyspark.enabled' is set to true; however, failed by the reason below:
'JavaPackage' object is not callable
Attempting non-optimization as 'spark.sql.execution.arrow.pyspark.fallback.enabled' is set to true.

Because of this, the conversion is very slow if it even finishes.
Is it a bug or can i do something about it?

Best regards

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
4,451 questions
Accepted answer
  1. Bush, Alex 86 Reputation points
    2022-07-05T20:57:58.207+00:00

    I was also getting this issue and raised a ticket with MS support. They acknowledged the issue and gave me the following workaround for the meantime:

    At the top of your code add the following lines: 

    from py4j.java_gateway import java_import   
java_import(spark._sc._jvm, "org.apache.spark.sql.api.python.*")

    For example:
    217877-screenshot-2022-07-06-085607.jpg

    2 people found this answer helpful.

1 additional answer

Sort by: Most helpful
Most helpful Newest Oldest
  1. PRADEEPCHEEKATLA-MSFT 79,141 Reputation points Microsoft Employee
    2022-05-30T05:30:11.237+00:00

    Hello @Marek, Phil ,

    It's looks like a Warning message and you are getting the excepted output.

    As per the repro, we ran the above query on the Apache Spark runtime 3.1 and worked as excepted without any warning message.

    206548-image.png

    Hope this will help. Please let us know if any further queries.

    ------------------------------

    • Please don't forget to click on 130616-image.png or upvote 130671-image.png button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
    • Want a reminder to come back and check responses? Here is how to subscribe to a notification
    • If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators