I have a folder in Blob storage and i am placing a file inside it. In my data flow source option I have set After completion -> delete file.
So once my data flow is completed, the file actually gets deleted, but along with that my Folder also gets deleted and changes in to a file with size of 0B.
Due to this, when the pipeline is running again, it throws an error that the specific folder is not found. Although we have a workaround where we are checking for the folder existence and then processing the file, its cumbersome in the stage where the project is being scaled up.
Please suggest a solution.