ADLS Folder Deletion Issue in ADF Pipeline

Question

ADLS Folder Deletion Issue in ADF Pipeline

Jake Atkins 20

I am encountering an issue while creating a pipeline in Azure Data Factory (ADF) to delete folders in Azure Data Lake Storage (ADLS). The pipeline includes a 'Get Metadata' activity to retrieve a list of folders to delete, but it fails with the following error:

Failed to execute delete activity with data source 'AzureBlobFS' and error 'Operation returned an invalid status code 'Conflict''. For details, please reference log file here.

The pipeline fails at the 'Delete Files' activity. Here are the details of the pipeline configuration:

The 'Get Metadata' activity uses a binary ADLS Gen 2 dataset with the 'childitems' argument and is filtered by last modified time using the expression @{addminutes(pipeline().TriggerTime, -10)}.
Filter:
- Items: using expression @activity('Get List of Files').output.childItems
  - Condition: using expression @and(equals(item().type, 'Folder'), not(contains(item().name, 'FOLDER NAME'))).
  - For Each:
    - Sequential = OFF
      - Items: using expression @activity('Filter Files').output.value
        
        Activities: Delete Files
        
        Delete Files:
        
        Same binary dataset as above
        
        Dataset directory set, Filename set using expression @item().name
        
        File path type: Currently set to 'File path in dataset', but I have tried different options, including wildcard, with the same issue.
        
        Recursively: ON

The pipeline fails when attempting to delete the targeted folders. Any insights on what might be causing this conflict would be greatly appreciated.

Below is a screenshot of the pipeline:

User's image

Ganesh Gurram 7,295 Reputation points Microsoft External Staff Moderator

2025-04-01T18:37:16.6266667+00:00

@Jake Atkins

Just checking in to see if the below answer helped. If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.

1 answer

Your answer

Ganesh Gurram 7,295 Reputation points Microsoft External Staff Moderator

2025-04-01T18:37:16.6266667+00:00

@Jake Atkins

Just checking in to see if the below answer helped. If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.

Answer 1

Hello Jake Atkins,

Thank you for posting your question in the Microsoft Q&A forum.

The "Conflict" (HTTP 409) error in your ADF pipeline occurs when the Delete activity fails to remove folders from Azure Data Lake Storage (ADLS) Gen2. This must be happening due to one of the following reasons:

Folder Not Empty & Recursive Delete Not Working

The "Conflict" error usually means ADLS Gen2 cannot delete a non-empty folder unless Recursive = true is explicitly set.

Even if you enabled Recursive Delete, there might be:

Open file handles (leases) are preventing deletion.
Permission issues (RBAC or ACLs blocking deletion).
Concurrent modifications (another process modifying files while deletion is attempted).

2. Incorrect Path Resolution in Delete Activity

If the dataset path and item().name are not correctly combined, ADF may attempt to delete an invalid path.

Example Issue:

Dataset path: container/folder/
item().name: subfolder/
Expected path: container/folder/subfolder/
Incorrect path: container/folder/subfolder (missing trailing / for folders).

3. Time-Based Filtering Race Condition

Your filter uses @addminutes (pipeline().TriggerTime, -10), which may exclude recently modified folders that are still being written to.
If another process (e.g., Synapse, Databricks) is actively writing files, ADLS may block deletion.

You may need to verify the below configurations to resolve the issue:

Ensure Recursive Delete is Properly Configured (test manually first)
Correct Path Format for Folders (trailing / for folders)
Add Error Handling & Retry Logic
Check Permissions - ADF Managed Identity must have Storage Blob Data Owner (RBAC) or Execute + Write + Delete (ACLs) on the folder
Test with a Simpler Pipeline - Try to delete one known empty folder first

Some Microsoft documentation may help to learn more about the above pointers:

https://learn.microsoft.com/en-us/azure/data-factory/delete-activity

https://learn.microsoft.com/en-us/azure/data-factory/data-factory-troubleshoot-guide

https://learn.microsoft.com/en-us/rest/api/storageservices/datalakestoragegen2/path/delete?view=rest-storageservices-datalakestoragegen2-2019-12-12#status-code-409

https://learn.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-directory-file-acl-powershell#delete-directories

If the above answer helped, please do not forget to "Accept Answer," as this may help other community members refer to the information if they face a similar issue. Your contribution to the Microsoft Q&A community is highly appreciated. Thank you for posting your question in the Microsoft Q&A forum.

Ganesh Gurram 7,295 Reputation points Microsoft External Staff Moderator

2025-04-02T18:13:39.6733333+00:00

@Anonymous

Following up to see if the above answer was helpful. If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.

Share via

ADLS Folder Deletion Issue in ADF Pipeline

1 answer

Your answer