Extra files in each folder when saving file from azure databricks to blob

Alexis Kan 21 Reputation points
2022-11-24T07:50:31.11+00:00

Hi,
I'm using databricks to mount to my blob storage with this function:

dbutils.fs.mount()

So I am fetching data and saving data from/to blob. I have partitioned my path so the path looks like <container_name>/year/month/day/<blob_file>. When I save my csv file it creates additional files in my partitions, that is /year/month/day. Below is a snapshot of how it looks like in folder month:

263768-image.png

Why is it creating those extra files and is it possible to avoid these additional files?

Azure Blob Storage
Azure Blob Storage
An Azure service that stores unstructured data in the cloud as blobs.
2,616 questions
Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,071 questions
0 comments No comments
{count} votes

Accepted answer
  1. ShaikMaheer-MSFT 38,326 Reputation points Microsoft Employee
    2022-11-25T08:37:57.213+00:00

    Hi @Alexis Kan ,

    Thank you for posting query in Microsoft Q&A Platform.

    Are you using blob end point here to mount? Please note, if we use blob then its expected behavior. Kindly check below link. Please share your code to understand more and try same at my end to help better.
    https://stackoverflow.com/questions/52872654/why-is-an-empty-file-with-the-name-of-folder-inside-a-azure-blob-storage-contain


0 additional answers

Sort by: Most helpful