Renaming a produced file in delta format - Synapse Analytics

Question

Renaming a produced file in delta format - Synapse Analytics

pmscorca 1,052

Hi, in Synapse Analytics I've implemented a data flow to read a csv source file and produced a sink file in delta format. If possible, I'd like to rename the produced parquet file in order to assign a meaningful name. Any suggests to me, please?
User's image

PRADEEPCHEEKATLA 90,641 Reputation points Moderator

2023-04-27T09:03:38.6733333+00:00

@pmscorca - Following up to see if the below answer was helpful. If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.

1 answer

Your answer

PRADEEPCHEEKATLA 90,641 Reputation points Moderator

2023-04-27T09:03:38.6733333+00:00

@pmscorca - Following up to see if the below answer was helpful. If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.

Answer 1

PRADEEPCHEEKATLA 90,641 Moderator

pmscorca - Thanks for the question and using MS Q&A platform.

You can use Microsoft Spark Utilities - to rename file.

mssparkutils.fs provides utilities for working with various file systems, including Azure Data Lake Storage Gen2 (ADLS Gen2) and Azure Blob Storage.

To demo practically, I had a part-00000 file in the delta folder as shown below:
User's image

Utilized mssparkutils.fs.mv command to rename the part-00000 file with the meaning full name as per your requirement.

old_name = r"abfss://******@cheprasynapse.dfs.core.windows.net/delta/delta-table-0/part-00000-dfd2415b-7a09-4e61-a8e5-96e9f38dc367-c000.snappy.parquet"
new_name = r"abfss://******@cheprasynapse.dfs.core.windows.net/delta/delta-table-0/pradeep.snappy.parquet"

mssparkutils.fs.mv(old_name, new_name)

User's image

We had successfully renamed part-00000 file with the meaning full name as per your requirement. User's image

Hope this helps. Do let us know if you any further queries.

pmscorca 1,052 Reputation points

2023-04-19T14:18:31.33+00:00

Hi, could I accomplish this task inside Synapse?
PRADEEPCHEEKATLA 90,641 Reputation points Moderator

2023-04-24T05:57:36.6466667+00:00

pmscorca - If you want to accomplish this task inside synapse pipeline, you need to create a new activity to read the parquetpart-00000 files generated and write to the file name as per your requirement.
OR simply, once the get the part-00000 files generated, you can call the above the notebook activity to rename the file name as per your requirement.

Hope this helps. Do let us know if you any further queries.

Share via

Renaming a produced file in delta format - Synapse Analytics

1 answer

Your answer