databricks - save to .parquet

arkiboys 9,696

Hello,
using pyspark, I run a select and then save to .parquet.
The problem is that it saves .parquet as well as othe rfiles such as _commited and _success, etc.
Question:
How can I change the pyspark to only save .parquet and have no other files?
Thanks

df = spark.sql('select * from viewName limit 100')
df.write.parquet('dbfs:/mnt/temp/foldername', mode='overwrite')

Saurabh Sharma 23,821 Reputation points Microsoft Employee

2022-03-04T00:54:16.44+00:00

Hi @arkiboys ,

Thanks for using Microsoft Q&A!!
As per my understanding you do not want to create additional files when saving a. parquet file using data bricks notebook.
I do not think that is possible, and you might want to delete the additional files after saving the parquet files instead of trying to avoid creating those additional files while saving. However, I am checking internally on the same and get back to you.

Thanks
Saurabh

Accepted answer

Costa , Ana 76 Reputation points

2022-03-31T13:00:41.8+00:00

I have the same problem.
For anybody looking for a quick fix meanwhile , I use this after creating a file with Python:

nameFile = [x.name for x in dbutils.fs.ls(f"{path}{fileName}.parquet") if x.name.split('.')[-1] == 'parquet'][0]
dbutils.fs.cp(f"{path}{fileName}.parquet/{nameFile}",f"{path}{fileName}.parquet")
dbutils.fs.rm(f"{path}{fileName}.parquet",recurse = True)
Please sign in to rate this answer.

0 comments No comments
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

Share via

databricks - save to .parquet

0 additional answers

Your answer