Azure Data Factory - Copy activity using REST connector - Pagination vs Retry

Question

Azure Data Factory - Copy activity using REST connector - Pagination vs Retry

Balachandran, Karthikeyan 25

Hi team,

I use ADF pipeline comprising copy activity to pull data from a remote endpoint using REST connector and then place the data on the blob. I use the pagination feature available with REST connector as the response is paginated.

I have a problem here - whenever retry is fired (say because of 50x at the source endpoint), retry does NOT resume from the call that failed but starts all over again from 1st page. How do I fix this? Please advise.

Thank you.

Balachandran, Karthikeyan 25 Reputation points

2023-08-14T11:41:46.0866667+00:00

Hi team,

Solicit response please. Am working with vendor API and we need a closure on this asap. Please help/guide me on this.
ShaikMaheer-MSFT 38,546 Reputation points Microsoft Employee Moderator

2023-08-18T04:39:58.85+00:00

Hi Balachandran, Karthikeyan, Just checking if any updates on above. If yes, please consider hitting Accept Answer button. Accepted answers help community as well. Please let me know if any further queries. Thank you.

1 answer

Your answer

Balachandran, Karthikeyan 25 Reputation points

2023-08-14T11:41:46.0866667+00:00

Hi team,

Solicit response please. Am working with vendor API and we need a closure on this asap. Please help/guide me on this.
ShaikMaheer-MSFT 38,546 Reputation points Microsoft Employee Moderator

2023-08-18T04:39:58.85+00:00

Hi Balachandran, Karthikeyan, Just checking if any updates on above. If yes, please consider hitting Accept Answer button. Accepted answers help community as well. Please let me know if any further queries. Thank you.

Answer 1

ShaikMaheer-MSFT 38,546 Microsoft Employee Moderator

Hi Balachandran, Karthikeyan,

Thank you for posting query in Microsoft Q&A Platform.

yes, you are right. Retry option will actually re-run entire activity again. That means copy activity will re-run all API calls again. So, in case of failure of copy activity if partial data already moved to blob, then don't consider using retry of copy activity. Instead, you can have different activities to run on failure of copy activity, to identify up to what page data loaded and loaded from there next pages.

Basically, it's like have an extension of process in the pipeline on failure of copy activity, to identify till what page data loaded and process next steps from there.
Hope this helps. Please let me know how it goes.

ShaikMaheer-MSFT 38,546 Reputation points Microsoft Employee Moderator

2023-08-16T10:59:44.8366667+00:00

Hi Balachandran, Karthikeyan, Just checking if any updates on above. If yes, please consider hitting Accept Answer button. Accepted answers help community as well. Please let me know if any further queries. Thank you.
Balachandran, Karthikeyan 25 Reputation points

2023-08-23T16:12:56.5366667+00:00
Hi @ShaikMaheer-MSFT / team,

Could you please be more specific?

(1) You mention "if partial data already moved to blob".

Copy Activity - that pulls from external endpoint and writes to blob - if there is a failure at some point, I see no file written to the blob. How do I write to blob whatever has been pulled so far? I observe it is only either full response (after all pages pulled) or no file.

(2) To add, how do I parse the output at the time of failure to identify the point at which failure occurred so that I may setup subsequent parts of the pipeline to retry from where it was left off?

I would request more pointed and clear directions based on your expert knowledge of how Copy Activity (using REST connector) would work when pulling from an API and writing to blob.

For instance - below is how the typical output looks like when error occurred. How do you think I can glean the information about submitted URL (like activity's output's key) to be able to submit for retries/resumptions?

{

"dataRead": 824422210, "dataWritten": 104857600, "filesWritten": 0, "sourcePeakConnections": 1, "sinkPeakConnections": 1, "rowsRead": 2142000, "rowsCopied": 2142000, "copyDuration": 3237, "throughput": 257.07, "logFilePath": "XXXX/XXX/logging/copyactivity-logs/XXX/XXX/", "errors": [ { "Code": 23354, "Message": "Failure happened on 'Source' side. ErrorCode=RestCallFailedWithServerError,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Rest call failed with server error, status code 503 ServiceUnavailable, please check your activity settings and remote server issues.\nRequest URL: https://XXX:443/XXX?next_token=12005225.\nResponse: {\"message\":\"Service Unavailable\"},Source=Microsoft.DataTransfer.ClientLibrary,'", "EventType": 0, "Category": 5, "Data": { "FailureInitiator": "Source" }, "MsgId": null, "ExceptionType": null, "Source": null, "StackTrace": null, "InnerEventInfos": [] } ],
Balachandran, Karthikeyan 25 Reputation points

2023-08-23T16:21:29.6933333+00:00

SO,

Copy Activity with REST connector cannot retry from where an error occurred for a paginated response! It starts all over from 1st page!

And when I try to get around this issue with the vanilla Web Activity, I face the issue of maximum response of 4 MB ( https://feedback.azure.com/d365community/idea/4fa9dce8-6d26-ec11-b6e6-000d3a4f032c)! Experience with ADF has been really painful, to be kind with my choice of words.

NOW,

Could you please suggest an alternative to Copy Activity + REST connector and to Web Activity - that can help me with my real business problem I have - to be able to pull from vendor API and write response to blob - using a no-code solution like ADF?
ShaikMaheer-MSFT 38,546 Reputation points Microsoft Employee Moderator

2023-08-28T16:04:16.5333333+00:00

Hi Balachandran, Karthikeyan, If case of pagination failures, I would suggest having a separate pipeline to process records. In this pipeline, you should first try to look blob data and understand till what page data loaded and then try to process API calls from that page in this pipeline.

Hope this helps. Please let me know how it goes.

Please consider hitting Accept Answer button. Accepted answers help community as well. Thank you.
Balachandran, Karthikeyan 25 Reputation points

2023-08-29T05:11:19.62+00:00

Hi @ShaikMaheer-MSFT ,

Maybe the point is not coming across to you.

Per my observation, when using Copy Activity + REST connector + write to blob, there is NO intermediate file generated. Either a complete file with all contents post navigating across all pages OR zero file if any pagination query fails - is what we get.

Please advise on how I could separately reprocess from where it was left off when there is no intermediate file even generated!
ShaikMaheer-MSFT 38,546 Reputation points Microsoft Employee Moderator

2023-08-30T11:00:05.1133333+00:00

Hi Balachandran, Karthikeyan, Oh got you. Sorry for the confusion. I got it now.

If thats the case, then we should always try to make API calls from staring again. Bcs ADF will not have any log or temp storage internally to hold the pages for which call was successful. So we should re-run all API call from starting. Its by design of product.

But I would encourage you to feel free to submit your feedback using below link and up-vote same. Internal team checks these feedback closely.

https://feedback.azure.com/d365community/forum/1219ec2d-6c26-ec11-b6e6-000d3a4f032c

Hope this helps. Please let me know if any further queries.

Please consider hitting Accept Answer button. Accepted answers help community as well.

Share via

Azure Data Factory - Copy activity using REST connector - Pagination vs Retry

1 answer

Your answer