How to exclude a single file when deleting files from blob location

Priya Jha 896 Reputation points
2022-08-18T10:29:48.147+00:00

I have some files placed at a blob location and after some transformations i want to delete all files from that location except for one Dummy file via recursive delete activity as shown below:

232362-microsoftteams-image.png

How to exclude a single file named Dummy.txt when deleting all files using recursively property of Delete activity?

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,639 questions
{count} votes

Accepted answer
  1. SSingh-MSFT 16,371 Reputation points Moderator
    2022-08-19T05:51:52.633+00:00

    Hi @Priya Jha ,

    Thanks for using Microsoft Q&A platform.

    Suppose we have a few files present inside blob location as shown below with dummy.txt file:

    232693-image.png

    As per the requirement, we want to delete all files from that location except the Dummy file.

    Following Community Expert @Subashri Vasudevan , third approach, we may achieve the result by below steps:
    Overall Pipeline would look like this:

    232761-image.png

    First, using Get Metadata activity we will get the Child Items from the blob location, settings as below:

    232694-image.png

    Dataset Connection:

    232762-image.png

    Secondly, we filter out the “dummy” file as below:

    232771-image.png

    Next, we will iterate through the location using ForEach:

    232781-image.png

    Inside ForEach , we will use Delete activity to get all the files deleted:

    232791-image.png

    Hope these steps will help. Please let us know if any further queries.

    ------------------------------

    • Please don't forget to click on 130616-image.png or upvote 130671-image.png button whenever the information provided helps you.
      Original posters help the community find answers faster by identifying the correct answer. Here is how
    • Want a reminder to come back and check responses? Here is how to subscribe to a notification
    1 person found this answer helpful.
    0 comments No comments

1 additional answer

Sort by: Most helpful
  1. Subashri Vasudevan 11,231 Reputation points
    2022-08-18T14:29:28.833+00:00

    Hi @Priya Jha ,

    Couple of options,

    1. If we want to exclude one dummy file, it's better to keep that file into a directory inside the location which gets deleted.
    2. Or else before deleting the files, copy the dummy file to a temporary location and after deletion copy it back to the original location.
    3. One more option would be to use a get meta data activity and list all child items (this includes dummy file too) and then use a filter activity to filter out the dummy file and loop thru the output of filter activity to delete files in parallel.

    Let us know if you have questions.


Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.