Azure Data Factory - Copy Activity: Error when REST API returns a 204 (No Content)

Sebastian Vatland Sonesen 1 Reputation point
2021-01-21T14:07:28.253+00:00

I am copying data from a REST API to blob storage. This is done inside a ForEach, so several calls are being made, all with different parameters. For some of the calls, the REST API might return a 204, which means that the API returns no data for that specific call.

The pipeline then run returns a NullReferenceException. There seems to be no way around this issue, and "skip incompatible rows" does not work.

The request is valid, and it works for most of the items in the ForEach, but if one call returns no data, the whole pipeline fails.

Error Message:
Failure happened on 'Source' side. ErrorCode=UserErrorHttpStatusCodeIndicatingFailure,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=The HttpStatusCode 400 indicates failure. "[NullReferenceException] Object reference not set to an instance of an object.",Source=Microsoft.DataTransfer.ClientLibrary,'

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,627 questions
{count} votes

2 answers

Sort by: Most helpful
  1. MartinJaffer-MSFT 26,236 Reputation points
    2021-01-25T22:05:50.47+00:00

    Hello @Sebastian Vatland Sonesen and welcome to Microsoft Q&A.

    If you would like to make the failure of the copy activity, not fail the pipeline, then add a wait activity after the copy activity connected by an on-failure dependency.

    If there is already an activity following the copy activity, do not add the wait activity. Simple change the dependency from on-success to on-completion.

    When an activity is followed only by an on-failure dependency, this indicates you expected the failure to happen, and whatever follows is intended to handle the situation. In this way the pipeline will not fail.

    Please let me know if this solves your problem, or if you want more explanation / help. Thank you.

    0 comments No comments

  2. Luca D'Elicio 1 Reputation point
    2022-02-10T21:09:18.847+00:00

    Hi i have the same problem.

    I'm reading a hyerarchical JSON but not all the nodes exists in every JSON.

    When the node it's not present the copy activity goes in error.

    But i don't want that the pipeline finish when i have an other error.

    There is a way to catch this error when the node doesn't exists?

    Thanks for the support


Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.