Incremental data ingestion approach for REST API as source and ADLS as sink

Hi All,

Good Day!!

I have a requirement where I need to implement incremental data ingestion for REST API as source and write the data as json format in ADLS.

Can any one guide me on how we can fullfill the incremental data ingestion patter form REST API as source through ADF copy activity pipeline.

Also would like to know, the respective REST API has more then 20K JSON response which need to be copy as part of Full load, However it is getting failed in midway without the copying the data even though I kept limit 5 and added pagination feature with next data element until it is null.

Thank you in anticipation!!

Regards,
Mahammad Khan

MartinJaffer-MSFT 26,011 Reputation points

2021-08-04T00:27:21.27+00:00

Hello @Abdulla Mahammad Khan Dawood and welcome to Microsoft Q&A.

An incremental load implies getting only the most recent or only the changed data, as opposed to everything. Whether your REST API source supports this or not, is something you must investigate.

Generally, the process of an incremental load would involve sending a parameter to the source query, which reflects when the sink was last updated. The source query then uses this to get only the records changed after. Is your sink set up to provide such a detail?

You mentioned your load is getting failed partway. What sort of error message are you getting?
Abdulla Mahammad Khan Dawood 186 Reputation points

2021-08-04T06:01:27.143+00:00

Hi Martin,

Good Day!!

I am getting below error when I am trying to copy bulk data from REST API source.

Failure happened on 'Source' side. ErrorCode=RestSourceCallFailed,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=The HttpStatusCode 502 indicates failure. Request URL: https://api-sandbox.argyle.io/v1/activities?cursor=cD0yMDIxLTA2LTIyKzEzJTNBMzMlM0EwNCUyQjAwJTNBMDA%3D&limit=200 Response payload:,Source=Microsoft.DataTransfer.ClientLibrary,'

Can you please help us how we can prevent this error, as we are tried running with 200 as limit size and also tried to reduced 10 as limit but still not working.

Thank you in anticipation!!

Regards,
Mahammad Khan
MartinJaffer-MSFT 26,011 Reputation points

2021-08-04T17:42:57.92+00:00

502, as I understand it, means the API you queried, is in turn getting an error from where IT fetches data from.

Is that URL something I can try to reproduce the issue with, or does it need authentication?
Abdulla Mahammad Khan Dawood 186 Reputation points

2021-08-05T05:29:14.607+00:00

Thank you Martin!!

Here is the URL which I am trying to ingestion data from REST API to ADLS, as it is sandbox environment.

https://api-sandbox.argyle.io/v1/activities

Absolute URL including the max limit: https://api-sandbox.argyle.io/v1/activities?limit=200

IURL has basic authentication which has username and pwd to connect.

authentication type : Basic authentication

username : b743985f555c4b3fbaa55fcd988b083e

IIncluded respective password in a text file as an attachment.120791-authenticationpwd.txt

120656-copy-pipeline-json.txt

Can you help me how we can fix, here I took a screen shot during copy activity which got failed in midway. What I understand at source it has huge volume of data. For you reference I have included pipeline copy activity json where we added pagination rules.

Appreciate your help.
HimanshuSinha-msft 19,376 Reputation points Microsoft Employee

2021-08-27T17:53:58.107+00:00

Hello @Abdulla Mahammad Khan Dawood ,
Thanks for the patience , unfortunately we are not getting the kind of response from the team here . if you have a support plan you may file a support ticket, else could you please send an email to azcommunity@microsoft.com with the below details, so that we can create a one-time-free support ticket for you to work closely on this matter.
Subscription ID:
Subject : Attn Himanshu
Please let me know once you have done the same.

Thanks
Himanshu