How to make this step fail

Question

How to make this step fail

King Java 790

I am trying to send out email notification when there is no file inside designated Azure File Share location.

This was my original logic, but it does not appear to be doing what I expected ADF to be doing.

Even when there is no file with criteria, the first step (Copy data) still finished as successful.

User's image

How do I make the first step Fail if there is no data with what criteria indicated?

Accepted answer

0 additional answers

Your answer

Answer 1

Subashri Vasudevan 11,231

Hello @King Java

If you are using a wildcard in the file name , even if there's no matching file, the copy activity will succeed. It is because , wild cards do not result in error when there's is no matching file.

If you are specifically looking for a file, you can mention that file name in the data set, for example, abc.txt. If the copy activity doesn't find such file, if will fail. (This holds good when you parameterize the file name portion in the data set too)

If you have to use a wild card in file name part, you can follow below steps.

Add a data set to be used with wildcard. In the dataset add a parameter called extension, which needs to be used in file name part. (my data set name is DelimitedText1)
Add get meta data activity and pass wild card name as follows pointing to the data set created in above step (Im passing *.txt to match any txt files in the ADLS, I am sure it can be used with File share too)

Next to get meta data activity, add an if condition to check the number of matching files brought by the get meta data activity. Please use the below expression in the if condition to check for files.

@greater(length(activity('Get Metadata1').output.childItems),0)

This will evaluate to true if there are matching files, and false if there are no matching files. In the false part of if condition, add a fail activity and give message - File Not found and Error Code 404 (or anything). This will make sure to fail the pipeline if there are no files. Next to this activity you can connect your web activity to send notification.

In the true part, you can have your copy activity.

I had no .txt files in my data lake, so my pipeline failed. See below,

Screenshot 2023-07-18 at 9.27.04 AM

Hope it helps. Please let us know if you have questions.

King Java 790 Reputation points

2023-07-18T21:20:05.75+00:00

@Subashri Vasudevan

Thank you for your help.

My current issue is on the "Get Metadata", I cannot make the scenario where it fails.

I am not able to make this setting (shown on the bottom) not showing data.

Also, I am not sure how to have the "extension" option to be displayed like yours.

When I debugged using just "Get Metadata", I always get data as an output.

I am having a hard time making the dataset with more criteria.

Do I have to go to dataset itself and make it more specific?
King Java 790 Reputation points

2023-07-18T21:26:08.27+00:00

@Subashri Vasudevan

Thank you for your comment.

I am having a hard time make the "Get Metadata" to fail.

I am not sure how to make the "extension" show up like your illustration.

This is what I have.

I don' think I should make the dataset itself to be specific (with criteria) to make the output fail, right?
Subashri Vasudevan 11,231 Reputation points

2023-07-19T02:01:19.6133333+00:00

Hi @King Java

Sorry for the confusion. The extension option is nothing but a parameter that I created in dataset itself. In the data set that your getmeta data activity uses, add a parameter called extension or filename or anything. Once you have a parameter in your dataset, you will see an option like mine from getmeta data activity.

And, again, this method is for passing wildcard file name like abc*.txt.

This will not make getmeta activity to fail. Where the above approach fails is inside the if condition false part. That too we forcefully fail the pipeline when there's no matching file available based on the wildcard we use in getmeta data activity.

Here is a video, for your reference. In the video we are using exists method to check for file, which is an alternate way of checking files. You can follow the steps in the video too.

Hope it helps. Please let us know for any help.
King Java 790 Reputation points

2023-07-19T21:11:05.17+00:00

@Subashri Vasdudevan Thank you for your help.

I watched the YouTube you referred and I used 'exists' for the logic.

I also noticed (from the video) that I have to specify the wildcard path from dataset itself first, otherwise, the logic will not fail. Is this right?

It is because on the "Get Metadata", it only has the section for Dataset. There is no option to put wildcard.

Share via

How to make this step fail

0 additional answers

Your answer