Azure Data Factory Issue with After completion in Source Options

Question

Hello All,

I have a folder in Blob storage and i am placing a file inside it. In my data flow source option I have set After completion -> delete file.

So once my data flow is completed, the file actually gets deleted, but along with that my Folder also gets deleted and changes in to a file with size of 0B.

Due to this, when the pipeline is running again, it throws an error that the specific folder is not found. Although we have a workaround where we are checking for the folder existence and then processing the file, its cumbersome in the stage where the project is being scaled up.

Please suggest a solution.
Thanks

Answer 1

Hi @Monica Manoharan ,

Thank you for posting query in Microsoft Q&A Platform.

Since, its blob storage if you delete all files from folder, automatically folder will also disappear.

Why Folder is disappearing?
In Blob storage there is no File system hierarchy. Every thing is file or blob there.
For example, If you have a blob like "adfdemo/inputdata/demo.csv". Then here actually entier path is part of file name only. But for users convenient GUI shows it as folder and file structure.

If you want to make sure you should have folders and file hierarchy, then consider using ADLS Gen2.

Hope this will help. Please let us know if any further queries.

------------------------------

• Please don't forget to click on or upvote button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
• Want a reminder to come back and check responses? Here is how to subscribe to a notification
• If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators