Azure Data Factory - Test/Filter/Ignore 404 responses

Dirk Sachse 106 Reputation points
2022-04-01T17:11:54.733+00:00

Hello,
I am extracting data from a REST API. Sometimes I get a 404 error / response. How can I test a URL for 404 before I send it to a copy activitiy?
Is there a way to catch that error? It is OK if it happens and I dont want the pipeline to fail because of it.

Thanks,
Dirk

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,509 questions
0 comments No comments
{count} votes

Accepted answer
  1. MartinJaffer-MSFT 26,011 Reputation points
    2022-04-04T17:03:23.957+00:00

    Hello @Dirk Sachse ,
    Thanks for the question and using MS Q&A platform.

    As we understand the ask here is multipart:

    1. How to test for an error from a REST source
    2. using the test, skip the copy activity
    3. Whether the test is success or fail, do not cause pipeline to fail
    4. possibly capture the error

    For the first point, testing for an error respose, I recommend trying the Web Activity. The setup is a little different from the Copy activity, and it does not do pagination, but it does make the response more accessible for later logic.

    For the second point, we can use the various dependency types to branch execution. This however gets complicated when we take the third point into account. This is because a pipeline fails when "there is a success dependency, and it is not run." Please reference the below image in the following explanation.

    189855-image.png

    Here we have our Web activity to test for error response. On failure, it goes to a (placeholder) Wait activity. The Copy activity will run only when BOTH conditions are met: 1) the test finishes 2) the Wait activity is NOT run. By using the completion dependency and combining with the skipped dependency, we can simulate the "success" dependency of the test without running into the issue of pipeline reporting a failure when the test fails.

    For the fourth point, capturing the error message, we will want to swap out the placeholder Wait activity with something else. A Set Variable if you want to just store the error message for another activity. If you are using SQL, a Store Proc could write the message. If you are doing the Function-App-sends-email thing, there is that. Or maybe just another Web activity to post. There are many possibilities.

    I would need to test to double check, but I think the expression is something like

    @activity("Test for 404").Error.message  
    

    Pease do let me if you have any queries.

    Thanks
    Martin


    • Please don't forget to click on 130616-image.png or upvote 130671-image.png button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
    • Want a reminder to come back and check responses? Here is how to subscribe to a notification
      • If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators
    1 person found this answer helpful.

0 additional answers

Sort by: Most helpful