Fabric Notebook: Conversion of int column from Pandas to Spark fails

Jörg Neulist 20 Reputation points
2024-04-17T11:46:04.7533333+00:00

For a data transformation task on Microsoft Fabric, I am using Pandas DataFrames (because of some missing features in the Spark version).

When trying to push the data to tables, I have to convert to Spark, which fails. The following code highlights the problem:

import numpy
import pandas as pd

df = pd.DataFrame(['id'] + [numpy.int64(i) for i in range(100)])
print(df.dtypes)
display(df)

The result is:

/opt/spark/python/lib/pyspark.zip/pyspark/sql/pandas/conversion.py:428: UserWarning: createDataFrame attempted Arrow optimization because 'spark.sql.execution.arrow.pyspark.enabled' is set to true; however, failed by the reason below: Expected bytes, got a 'numpy.int64' object Attempting non-optimization as 'spark.sql.execution.arrow.pyspark.fallback.enabled' is set to true.

The code fails. If I remove the cast to int64, the error still appears, but the code is able to recover.

I found an older instance of the same bug here: https://learn.microsoft.com/en-us/answers/questions/852534/arrow-optimization-in-python-notebook-fails

The accepted resolution in that thread does not resolve the problem for me. Any suggestions?

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
4,973 questions
Microsoft Fabric Training
Microsoft Fabric Training
Microsoft Fabric: A Microsoft unified data platform.Training: Instruction to develop new skills.
46 questions
0 comments No comments
{count} votes

Accepted answer
  1. Smaran Thoomu 16,555 Reputation points Microsoft Vendor
    2024-04-17T18:31:17.9333333+00:00

    Hi @Jörg Neulist

    Thanks for the question and using MS Q&A platform.

    Are you referring to any Microsoft Fabric Training learning module or your question specific to Microsoft Fabric product?

    MS Fabric is currently not supported in the Q&A forums, the supported products are listed over here Supported products on MS Q&A (more to be added later on).

    You can ask the experts in the dedicated MS Fabric community forum.

    Can you please open as a new thread in MS Fabric community: https://community.fabric.microsoft.com/t5/General-Discussion/bd-p/ac_generaldiscussion

    User's image

    Hope this helps. Do let us know if you any further queries.


    If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.

    0 comments No comments

0 additional answers

Sort by: Most helpful

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.