Azure Data Factory - Test/Filter/Ignore 404 responses

Question

Hello,
I am extracting data from a REST API. Sometimes I get a 404 error / response. How can I test a URL for 404 before I send it to a copy activitiy?
Is there a way to catch that error? It is OK if it happens and I dont want the pipeline to fail because of it.

Thanks,
Dirk

Answer 1

Hello @Dirk Sachse ,
Thanks for the question and using MS Q&A platform.

As we understand the ask here is multipart:

1. How to test for an error from a REST source
2. using the test, skip the copy activity
3. Whether the test is success or fail, do not cause pipeline to fail
4. possibly capture the error

For the first point, testing for an error respose, I recommend trying the Web Activity. The setup is a little different from the Copy activity, and it does not do pagination, but it does make the response more accessible for later logic.

For the second point, we can use the various dependency types to branch execution. This however gets complicated when we take the third point into account. This is because a pipeline fails when "there is a success dependency, and it is not run." Please reference the below image in the following explanation.

Here we have our Web activity to test for error response. On failure, it goes to a (placeholder) Wait activity. The Copy activity will run only when BOTH conditions are met: 1) the test finishes 2) the Wait activity is NOT run. By using the completion dependency and combining with the skipped dependency, we can simulate the "success" dependency of the test without running into the issue of pipeline reporting a failure when the test fails.

For the fourth point, capturing the error message, we will want to swap out the placeholder Wait activity with something else. A Set Variable if you want to just store the error message for another activity. If you are using SQL, a Store Proc could write the message. If you are doing the Function-App-sends-email thing, there is that. Or maybe just another Web activity to post. There are many possibilities.

I would need to test to double check, but I think the expression is something like

@activity("Test for 404").Error.message  

Pease do let me if you have any queries.

Thanks
Martin

---

• Please don't forget to click on or upvote button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
• Want a reminder to come back and check responses? Here is how to subscribe to a notification
  • If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators