Azure Data Factory - Ignore Errors

Question

Azure Data Factory - Ignore Errors

bert 5

Hi

I have an ADF Pipeline that uses a delete activity to delete files from Azure File Share storage.

I provide a list of files that need deleting to the pipeline, then it deletes them.

The problem I am having is, if the list of files contains folders that don't exist the delete activity fails.

In the log file I can see that after 9 instances of the error "The specified parent path does not exist." the pipeline stops running.

In my file there will be about 200,000 file paths listed and about half of them will not exist. Is there any way I can get the pipeline to ignore these errors and continue?

Bhargava-MSFT 31,261 Reputation points Microsoft Employee Moderator

2023-07-20T20:38:53.5466667+00:00
Hello bert,

Welcome to the Microsoft Q&A forum.

Please correct me if my understanding is wrong. You are looking to iterate through the folders and delete files from the folders,

If this is correct, please follow the below steps.

Add a Get Metadata activity to your pipeline and configure it to point to the subfolder's root folder.

In the Get Metadata activity, add a field to retrieve the list of subfolders by setting the field to "Child Items"

Add a ForEach activity to your pipeline and connect the Get Metadata activity to the ForEach activity.

Set the items of the ForEach activity to @activity('GetMetadataActivity1').output.childItems

Add your Delete activity to delete the file in the True branch of the If Condition activity.

In the False, you can leave it empty or add any activity you want to execute when the file does not exist.

I hope this helps. Please let me know if you have any further questions.
Bhargava-MSFT 31,261 Reputation points Microsoft Employee Moderator

2023-07-20T20:45:41.72+00:00

To Delete contents in the folder and the folder itself, we need to check the recursively option.

Please follow the below blog for more details:

https://www.sqlservercentral.com/articles/working-with-the-delete-activity-in-azure-data-factory
Bhargava-MSFT 31,261 Reputation points Microsoft Employee Moderator

2023-07-25T15:30:08.2933333+00:00

Hello bert,

I am checking to see if you have any further questions here.

If this answers your question, please consider accepting the answer by hitting the Accept answer and up-vote as it helps the community look for answers to similar questions.
bert 5 Reputation points

2023-07-26T08:22:32.8533333+00:00

Thanks for this, however I'm not sure this is exactly what I need.

I have a file called 'list.txt' that lists the full file-paths of the files that need deleting. I'm not sure on this setup where the list of files is being read.

I'm not trying to delete every file that exists in the File Share, but every file that is listed on 'list.txt', half of which are in File Share storage and the other half in blob storage.

For example there are 2 million files on the File Share, I have a list of 200,000 files that I need to delete. Of these 200,000 files 100,000 are on the File Share, and 100,000 are on the Blob.

The problem I'm having is that after the pipeline encounters just 9 file paths that aren't on the File Share, it stops running, I need it to carry on.

Strangely I do not get this issue when running the delete activity against Blob Storage, errors for missing parent path seem to get ignored.
Bhargava-MSFT 31,261 Reputation points Microsoft Employee Moderator

2023-07-26T19:14:25.8166667+00:00

bert

Thanks for providing the additional details in your latest comment.

Sorry, I didn't know you read files through the "list of files" option in the delete activity.

I tried to reproduce the issue from my end. but I see the same behavior(success) with the blob and file system.

I didn't get any failures if the file path didn't exist on the "file system" text file.

Were you able to delete any other files in the file system(with the delete activity)? or is it failing for all of the files?

It seems like the issue is with the file system-linked service configuration.

Did you point the linked service configuration to the list.txt file?
Bhargava-MSFT 31,261 Reputation points Microsoft Employee Moderator

2023-07-27T13:45:49.6866667+00:00

bert

I am checking to see if you got a chance to look into my above response.
bert 5 Reputation points

2023-07-27T14:35:19.92+00:00

I removed all instance of files that aren't present on the FileShare in a new list and the pipeline completed successfully without error, deleting all files it was supposed to.

In additional testing, I have provided a list of 13 files, 12 of which point files that don't exist in folders that don't exist, and 1 points to a file that does exist.

If the file that exists is at the top of the list, it gets deleted, then the activity goes on to error 9 times before the pipeline fails.

If the file that exists is at the bottom of the list, the delete activity errors on the first 9 then the pipeline fails before deleting the file that exists.

The dataset points to clientlettersuat/input and this is where the list.txt is stored

The list then includes the relative paths of the files to be deleted with respect to this location, so they need to go up a folder first as the files are stored in locations like this:

clientlettersuat/test/test001.txt

clientlettersuat/test2/text002.txt

which require relative paths from clientlettersuat/input that look like this:

../test/test001.txt

../test2/text002.txt

T

Bhargava-MSFT 31,261 Microsoft Employee Moderator

bert

Thanks for the additional details. I was able to reproduce the issue from my end. But I was getting a different error. "Could not find a part of the path."

This could be an expected behavior. I will check with the PG and get back to you with more details.

Here is my text file:

\test\13.txt

1.txt

2.txt

/test2/test3/3.txt

/test/4.txt

4.txt

Error message:

Name,Category,Status,Error
/test2/test3/3.txt,File,Failed,Could not find a part of the path 'D:\adf\new\1/test2/test3/3.txt'.
/test2/test3/3.txt,File,Failed,Could not find a part of the path 'D:\adf\new\1/test2/test3/3.txt'.
/test2/test3/3.txt,File,Failed,Could not find a part of the path 'D:\adf\new\1/test2/test3/3.txt'.
2.txt,File,Deleted,
/test2/test3/3.txt,File,Failed,Could not find a part of the path 'D:\adf\new\1/test2/test3/3.txt'.
/test2/test3/3.txt,File,Failed,Could not find a part of the path 'D:\adf\new\1/test2/test3/3.txt'.
/test2/test3/3.txt,File,Failed,Could not find a part of the path 'D:\adf\new\1/test2/test3/3.txt'.
\test\13.txt,File,Deleted,
1.txt,File,Deleted,
/test2/test3/3.txt,File,Failed,Could not find a part of the path 'D:\adf\new\1/test2/test3/3.txt'.
/test2/test3/3.txt,File,Failed,Could not find a part of the path 'D:\adf\new\1/test2/test3/3.txt'.
/test2/test3/3.txt,File,Failed,Could not find a part of the path 'D:\adf\new\1/test2/test3/3.txt'.

Bhargava-MSFT 31,261 Reputation points Microsoft Employee Moderator

2023-08-01T20:19:12.5366667+00:00

Hello bert,

I tried reaching out to my internal team for further clarification on the delete activity behavior for the filesystem. But I am not getting a response. I would suggest opening a support case for this issue. In case you don't have a support plan, please let me know so that I can enable a one-time free support ticket to work on this issue.

I am looking forward to hearing from you.
bert 5 Reputation points

2023-08-09T08:06:35.1933333+00:00

Thank you, I do not have a support plan so could enable the one-time support ticket please?
Bhargava-MSFT 31,261 Reputation points Microsoft Employee Moderator

2023-08-09T16:40:42.0666667+00:00

Hello bert,

Could you please send an email to AzCommunity@microsoft.com with the below details so that we can enable one-time-free support for you?

Email subject: <Attn - Bhargava : Microsoft Q&A Thread title>

Thread URL: <Microsoft Q&A Thread>

Subscription ID: <your subscription id>

Looking forward to your reply.

Regards,

BhargavaGunnam-MSFT

AzCommunity@microsoft.com

Your answer

Bhargava-MSFT 31,261 Reputation points Microsoft Employee Moderator

2023-07-20T20:38:53.5466667+00:00

Hello bert,

Welcome to the Microsoft Q&A forum.

Please correct me if my understanding is wrong. You are looking to iterate through the folders and delete files from the folders,

If this is correct, please follow the below steps.

Add a Get Metadata activity to your pipeline and configure it to point to the subfolder's root folder.

In the Get Metadata activity, add a field to retrieve the list of subfolders by setting the field to "Child Items"

Add a ForEach activity to your pipeline and connect the Get Metadata activity to the ForEach activity.

Set the items of the ForEach activity to @activity('GetMetadataActivity1').output.childItems

Add your Delete activity to delete the file in the True branch of the If Condition activity.

In the False, you can leave it empty or add any activity you want to execute when the file does not exist.

I hope this helps. Please let me know if you have any further questions.
Bhargava-MSFT 31,261 Reputation points Microsoft Employee Moderator

2023-07-20T20:45:41.72+00:00

To Delete contents in the folder and the folder itself, we need to check the recursively option.

Please follow the below blog for more details:

https://www.sqlservercentral.com/articles/working-with-the-delete-activity-in-azure-data-factory
Bhargava-MSFT 31,261 Reputation points Microsoft Employee Moderator

2023-07-25T15:30:08.2933333+00:00

Hello bert,

I am checking to see if you have any further questions here.

If this answers your question, please consider accepting the answer by hitting the Accept answer and up-vote as it helps the community look for answers to similar questions.
bert 5 Reputation points

2023-07-26T08:22:32.8533333+00:00

Thanks for this, however I'm not sure this is exactly what I need.

I have a file called 'list.txt' that lists the full file-paths of the files that need deleting. I'm not sure on this setup where the list of files is being read.

I'm not trying to delete every file that exists in the File Share, but every file that is listed on 'list.txt', half of which are in File Share storage and the other half in blob storage.

For example there are 2 million files on the File Share, I have a list of 200,000 files that I need to delete. Of these 200,000 files 100,000 are on the File Share, and 100,000 are on the Blob.

The problem I'm having is that after the pipeline encounters just 9 file paths that aren't on the File Share, it stops running, I need it to carry on.

Strangely I do not get this issue when running the delete activity against Blob Storage, errors for missing parent path seem to get ignored.
Bhargava-MSFT 31,261 Reputation points Microsoft Employee Moderator

2023-07-26T19:14:25.8166667+00:00

bert

Thanks for providing the additional details in your latest comment.

Sorry, I didn't know you read files through the "list of files" option in the delete activity.

I tried to reproduce the issue from my end. but I see the same behavior(success) with the blob and file system.

I didn't get any failures if the file path didn't exist on the "file system" text file.

Were you able to delete any other files in the file system(with the delete activity)? or is it failing for all of the files?

It seems like the issue is with the file system-linked service configuration.

Did you point the linked service configuration to the list.txt file?
Bhargava-MSFT 31,261 Reputation points Microsoft Employee Moderator

2023-07-27T13:45:49.6866667+00:00

bert

I am checking to see if you got a chance to look into my above response.
bert 5 Reputation points

2023-07-27T14:35:19.92+00:00

I removed all instance of files that aren't present on the FileShare in a new list and the pipeline completed successfully without error, deleting all files it was supposed to.

In additional testing, I have provided a list of 13 files, 12 of which point files that don't exist in folders that don't exist, and 1 points to a file that does exist.

If the file that exists is at the top of the list, it gets deleted, then the activity goes on to error 9 times before the pipeline fails.

If the file that exists is at the bottom of the list, the delete activity errors on the first 9 then the pipeline fails before deleting the file that exists.

The dataset points to clientlettersuat/input and this is where the list.txt is stored

The list then includes the relative paths of the files to be deleted with respect to this location, so they need to go up a folder first as the files are stored in locations like this:

clientlettersuat/test/test001.txt

clientlettersuat/test2/text002.txt

which require relative paths from clientlettersuat/input that look like this:

../test/test001.txt

../test2/text002.txt

T
Bhargava-MSFT 31,261 Reputation points Microsoft Employee Moderator

2023-07-28T02:25:02.5266667+00:00

bert

Thanks for the additional details. I was able to reproduce the issue from my end. But I was getting a different error. "Could not find a part of the path."

This could be an expected behavior. I will check with the PG and get back to you with more details.

Here is my text file:

\test\13.txt

1.txt

2.txt

/test2/test3/3.txt

/test2/test3/3.txt

/test2/test3/3.txt

/test2/test3/3.txt

/test2/test3/3.txt

/test2/test3/3.txt

/test2/test3/3.txt

/test2/test3/3.txt

/test2/test3/3.txt

/test2/test3/3.txt

/test2/test3/3.txt

/test2/test3/3.txt

/test/4.txt

4.txt

Error message:

Name,Category,Status,Error /test2/test3/3.txt,File,Failed,Could not find a part of the path 'D:\adf\new\1/test2/test3/3.txt'. /test2/test3/3.txt,File,Failed,Could not find a part of the path 'D:\adf\new\1/test2/test3/3.txt'. /test2/test3/3.txt,File,Failed,Could not find a part of the path 'D:\adf\new\1/test2/test3/3.txt'. 2.txt,File,Deleted, /test2/test3/3.txt,File,Failed,Could not find a part of the path 'D:\adf\new\1/test2/test3/3.txt'. /test2/test3/3.txt,File,Failed,Could not find a part of the path 'D:\adf\new\1/test2/test3/3.txt'. /test2/test3/3.txt,File,Failed,Could not find a part of the path 'D:\adf\new\1/test2/test3/3.txt'. \test\13.txt,File,Deleted, 1.txt,File,Deleted, /test2/test3/3.txt,File,Failed,Could not find a part of the path 'D:\adf\new\1/test2/test3/3.txt'. /test2/test3/3.txt,File,Failed,Could not find a part of the path 'D:\adf\new\1/test2/test3/3.txt'. /test2/test3/3.txt,File,Failed,Could not find a part of the path 'D:\adf\new\1/test2/test3/3.txt'.
Bhargava-MSFT 31,261 Reputation points Microsoft Employee Moderator

2023-08-01T20:19:12.5366667+00:00

Hello bert,

I tried reaching out to my internal team for further clarification on the delete activity behavior for the filesystem. But I am not getting a response. I would suggest opening a support case for this issue. In case you don't have a support plan, please let me know so that I can enable a one-time free support ticket to work on this issue.

I am looking forward to hearing from you.
bert 5 Reputation points

2023-08-09T08:06:35.1933333+00:00

Thank you, I do not have a support plan so could enable the one-time support ticket please?
Bhargava-MSFT 31,261 Reputation points Microsoft Employee Moderator

2023-08-09T16:40:42.0666667+00:00

Hello bert,

Could you please send an email to AzCommunity@microsoft.com with the below details so that we can enable one-time-free support for you?

Email subject: <Attn - Bhargava : Microsoft Q&A Thread title>

Thread URL: <Microsoft Q&A Thread>

Subscription ID: <your subscription id>

Looking forward to your reply.

Regards,

BhargavaGunnam-MSFT

AzCommunity@microsoft.com

Share via

Azure Data Factory - Ignore Errors

Your answer