Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
4,929 questions
Pandas 'df.to_parquet' stopped working
Jaryd
177
Reputation points
Hello, I have a notebook in synapse which was running just fine on a schedule from a Pipeline.
It just started to fail without any changes and I don't know why.
I was able to work around by saving the pandas dataframe to a spark dataframe and saving parquet from that, but this isn't really what I was hoping for.
df.to_parquet('abfs://root@spacecontainer.dfs.core.windows.net/Folder1/' + 'filename.parquet')
]1
{count} votes
-
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(134.4, 9%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKM%3C/text%3E%3C/svg%3E)
KranthiPakala-MSFT 46,492 Reputation points Microsoft Employee2022-02-18T21:26:48.513+00:00 Hello @Jaryd ,
Thanks for the question and using MS Q&A platform.
As I understand the ask here is that Pandas 'df.to_parquet' was working fine until yesterday and all of sudden it started failing with above error. Please correct me if I'm not accurate.
This looks like there is something changed which is causing this issue. To further troubleshoot the issue, could you please file a support ticket and do share the SR# with us for tracking it. In case if you don't have a support plan, do let us know here.
-
Jaryd 177 Reputation points
2022-02-18T21:30:05.897+00:00 Hello, this is on my personal account and I don't have a support plan.
I have currently circumvented the issue by writing the padnas DF to a spark DF and then saving.
-
Jaryd 177 Reputation points
2022-02-25T15:44:27.597+00:00 I have further resolved this by eliminating the needs for pandas all together and just handling everything in Pyspark. This was the best route, but still strange that the padnas df stopped saving. Doing some reading along the way, I see there is a Koalas library made for pyspark as a Pandas replacement. That would have been my next step.
Sign in to comment
Sign in to answer