question

MonicaManoharan-9413 avatar image
0 Votes"
MonicaManoharan-9413 asked ShaikMaheer-MSFT commented

Azure Data Factory Issue with After completion in Source Options

Hello All,

I have a folder in Blob storage and i am placing a file inside it. In my data flow source option I have set After completion -> delete file.

So once my data flow is completed, the file actually gets deleted, but along with that my Folder also gets deleted and changes in to a file with size of 0B.

Due to this, when the pipeline is running again, it throws an error that the specific folder is not found. Although we have a workaround where we are checking for the folder existence and then processing the file, its cumbersome in the stage where the project is being scaled up.

Please suggest a solution.
Thanks

azure-data-factory
· 2
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hi @MonicaManoharan-9413 ,

Just checking in to see if the below answer helped. If this answers your query, do click 130616-image.png and upvote 130671-image.png for the same. And, if you have any further query do let us know.

0 Votes 0 ·

Hi @MonicaManoharan-9413 ,

Following to see if the below answer helped. If this answers your query, do click 130616-image.png and upvote 130671-image.png for the same. And, if you have any further query do let us know.

0 Votes 0 ·

1 Answer

ShaikMaheer-MSFT avatar image
1 Vote"
ShaikMaheer-MSFT answered

Hi @MonicaManoharan-9413 ,

Thank you for posting query in Microsoft Q&A Platform.

Since, its blob storage if you delete all files from folder, automatically folder will also disappear.

Why Folder is disappearing?
In Blob storage there is no File system hierarchy. Every thing is file or blob there.
For example, If you have a blob like "adfdemo/inputdata/demo.csv". Then here actually entier path is part of file name only. But for users convenient GUI shows it as folder and file structure.

If you want to make sure you should have folders and file hierarchy, then consider using ADLS Gen2.

Hope this will help. Please let us know if any further queries.


  • Please don't forget to click on 130616-image.png or upvote 130671-image.png button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how

  • Want a reminder to come back and check responses? Here is how to subscribe to a notification

  • If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators




5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.