mssparkutils.fs.mv doesn't create new folder if it doesn't exists

Question

mssparkutils.fs.mv doesn't create new folder if it doesn't exists

Raghul Kannan 191

Hi,

I'm using

mssparkutils.fs.mv("abfss://******@storageaccount.dfs.core.net/folder1/test.csv","abfss://******@storageaccount.dfs.core.net/folder2",True) to move a file from folder1 to folder2. However, folder2 isn't getting created if it doesn't exist. For some reason, "file" called "folder2" is created.

Bhargava-MSFT 31,261 Reputation points Microsoft Employee Moderator

2023-07-26T20:35:13.0266667+00:00
Hello Raghul Kannan,

I see the same behavior from my end. I will check with my internal team and get back to you.

Meanwhile to copy files from a folder, please use the folder path directly. I was able to move a file from folder1 to folder3 (but it will copy all files)

mssparkutils.fs.mv('abfss://******@bhargavasynapsegen2.dfs.core.windows.net/folder1/','abfss://******@bhargavasynapsegen2.dfs.core.windows.net/folder3/',True)

As per the documentation, the command supported copying your test.csv to folder2

mssparkutils.fs.mv:

Moves a file or directory. Supports move across file systems.

PythonCopy

mssparkutils.fs.mv('source file or directory', 'destination directory', True) # Set the last parameter as True to firstly create the parent directory if it does not exist

https://learn.microsoft.com/en-us/azure/synapse-analytics/spark/microsoft-spark-utilities?pivots=programming-language-python#move-file
Raghul Kannan 191 Reputation points

2023-07-27T04:05:43.79+00:00

The suggestion you made won't work as I'm moving file to folder. Please note that this was working sometime back (not sure when it stopped working) and it's a breaking change/bug
Bhargava-MSFT 31,261 Reputation points Microsoft Employee Moderator

2023-07-27T14:33:03.79+00:00

Hello Raghul Kannan,

Yes, If you are moving file to the folder, it won't work. I am checking with my internal team and get back to you as soon as I hear something from them.

Accepted answer

0 additional answers

Your answer

Bhargava-MSFT 31,261 Reputation points Microsoft Employee Moderator

2023-07-26T20:35:13.0266667+00:00

Hello Raghul Kannan,

I see the same behavior from my end. I will check with my internal team and get back to you.

Meanwhile to copy files from a folder, please use the folder path directly. I was able to move a file from folder1 to folder3 (but it will copy all files)

mssparkutils.fs.mv('abfss://******@bhargavasynapsegen2.dfs.core.windows.net/folder1/','abfss://******@bhargavasynapsegen2.dfs.core.windows.net/folder3/',True)

As per the documentation, the command supported copying your test.csv to folder2

mssparkutils.fs.mv:

Moves a file or directory. Supports move across file systems.

PythonCopy

mssparkutils.fs.mv('source file or directory', 'destination directory', True) # Set the last parameter as True to firstly create the parent directory if it does not exist

https://learn.microsoft.com/en-us/azure/synapse-analytics/spark/microsoft-spark-utilities?pivots=programming-language-python#move-file
Raghul Kannan 191 Reputation points

2023-07-27T04:05:43.79+00:00

The suggestion you made won't work as I'm moving file to folder. Please note that this was working sometime back (not sure when it stopped working) and it's a breaking change/bug
Bhargava-MSFT 31,261 Reputation points Microsoft Employee Moderator

2023-07-27T14:33:03.79+00:00

Hello Raghul Kannan,

Yes, If you are moving file to the folder, it won't work. I am checking with my internal team and get back to you as soon as I hear something from them.

Answer 1

Bhargava-MSFT 31,261 Microsoft Employee Moderator

Hello Raghul Kannan,

<update>

PG confirmed this is expected behavior. Since the mv API treats mv test.csv to another file folder2.

You can either use the "destination file path" or make sure folder2 already exists or use mv API like below.

mssparkutils.fs.mv("abfss://******@storageaccount.dfs.core.net/folder1/test.csv","abfss://******@storageaccount.dfs.core.net/folder2/test.csv",True)

PG is going to update the documentation: ‘destination directory’, to "destination file path" on the below command.

mssparkutils.fs.mv('source file or directory', 'destination directory', True) # Set the last parameter as True to firstly create the parent directory if it does not exist

I hope this answers your question.

Please let me know if you have any further questions.

Bhargava-MSFT 31,261 Reputation points Microsoft Employee Moderator

2023-08-01T20:41:10.1233333+00:00

Hello Raghul Kannan,

I am checking to see if you got a chance to look into my above answer. Please let me know if you have any further questions.
Raghul Kannan 191 Reputation points

2023-08-02T00:49:49.27+00:00

Hi,

Yes. Thanks for the update
Bhargava-MSFT 31,261 Reputation points Microsoft Employee Moderator

2023-08-02T14:27:16.9266667+00:00

Thank you, Raghul Kannan

Share via

mssparkutils.fs.mv doesn't create new folder if it doesn't exists

0 additional answers

Your answer