Fabric Notebook: Conversion of int column from Pandas to Spark fails

Question

For a data transformation task on Microsoft Fabric, I am using Pandas DataFrames (because of some missing features in the Spark version).

When trying to push the data to tables, I have to convert to Spark, which fails. The following code highlights the problem:

import numpy
import pandas as pd

df = pd.DataFrame(['id'] + [numpy.int64(i) for i in range(100)])
print(df.dtypes)
display(df)

The result is:

/opt/spark/python/lib/pyspark.zip/pyspark/sql/pandas/conversion.py:428: UserWarning: createDataFrame attempted Arrow optimization because 'spark.sql.execution.arrow.pyspark.enabled' is set to true; however, failed by the reason below: Expected bytes, got a 'numpy.int64' object Attempting non-optimization as 'spark.sql.execution.arrow.pyspark.fallback.enabled' is set to true.

The code fails. If I remove the cast to int64, the error still appears, but the code is able to recover.

I found an older instance of the same bug here: https://learn.microsoft.com/en-us/answers/questions/852534/arrow-optimization-in-python-notebook-fails

The accepted resolution in that thread does not resolve the problem for me. Any suggestions?

Answer 1

Hi @Jörg Neulist

Thanks for the question and using MS Q&A platform.

Are you referring to any Microsoft Fabric Training learning module or your question specific to Microsoft Fabric product?

MS Fabric is currently not supported in the Q&A forums, the supported products are listed over here Supported products on MS Q&A (more to be added later on).

You can ask the experts in the dedicated MS Fabric community forum.

Can you please open as a new thread in MS Fabric community: https://community.fabric.microsoft.com/t5/General-Discussion/bd-p/ac_generaldiscussion

Hope this helps. Do let us know if you any further queries.

---

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.