Hi I am working in Azure Data Factory and doing "Copy Data Activity from HTTP Connector". It works for me but only the first page is getting saved into the database table. My question is how to do pagination in HTTP connector.

Hello @Wadhwa, Dinesh and thank you for your question. Pagination is tricky to implement in Azure Data Factory, currently. Before jumping into implementing a solution, I would like to let you know a few things. The HTTP connector does not have any pagination built-in. The REST connector has very limited pagination ability . You can create a feature request to add pagination to HTTP Connector, or upvote an existing feature request at the feedback forum. Depending upon the pagination mechanism your endpoint uses, we can do a work-around. Link to a related, but not exact workaround . The work-around involves looping over the page numbers, each a separate copy action. Since your sink is a database, the work-around is easier to implement because we do not need to worry about overwriting files. Please let me know which path you want to take (REST, HTTP loop work-around, feature request, or other). If you would like me to assist you in crafting a solution, please tell me how your website paginates. Thank you for your patience, Martin

Azure Data Factory - HTTP Connectors

Accepted answer

MartinJaffer-MSFT 26,076 Reputation points

2020-08-17T23:41:05.623+00:00

Hello @Wadhwa, Dinesh and thank you for your question. Pagination is tricky to implement in Azure Data Factory, currently. Before jumping into implementing a solution, I would like to let you know a few things.

The HTTP connector does not have any pagination built-in.
The REST connector has very limited pagination ability.

You can create a feature request to add pagination to HTTP Connector, or upvote an existing feature request at the feedback forum.

Depending upon the pagination mechanism your endpoint uses, we can do a work-around. Link to a related, but not exact workaround.

The work-around involves looping over the page numbers, each a separate copy action. Since your sink is a database, the work-around is easier to implement because we do not need to worry about overwriting files.

Please let me know which path you want to take (REST, HTTP loop work-around, feature request, or other). If you would like me to assist you in crafting a solution, please tell me how your website paginates.

Thank you for your patience,
Martin
Please sign in to rate this answer.
Wadhwa, Dinesh 21 Reputation points

2020-08-18T04:33:53.867+00:00

Hello @MartinJaffer-MSFT ,

Thanks for the reply.

I cannot use REST Connector due to an issue with Azure Data Factory as that Connector ignores Accept parameter and the API which I am using is expecting Accept: application/hal+json. ADF only use application/json as default. So due to this Azure data factory issue, I can't use this REST Connector approach.

I need to go with HTTP Connector. Now in the HTTP Connector:
1. I need to implement paging so that all the records from all pages should be saved into the database.
2. The auto-create table option is not working as it shows type(null) from JSON and it is giving some issue in Sink as well.

Wadhwa, Dinesh 21 Reputation points

2020-08-18T04:49:29.1+00:00

Also while using auto-create table option i get the below error. NO other details about the error. Please guide me.
{
"errorCode": "1000",
"message": "ErrorCode=SystemErrorStageDefTypeNotMatch,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=An internal error occurred when building stages for transfer. The input and output type of stage definition do not match.,Source=Microsoft.DataTransfer.ClientLibrary,'",
"failureType": "SystemError",
"target": "Copy data from HTTP Connector",
"details": []
}

MartinJaffer-MSFT 26,076 Reputation points

2020-08-18T18:26:14.233+00:00

Thank you for letting me know about this additional difficulty.
This error can be caused by a nested JSON record, especially if schemas have not been imported in the mapping step of copy activity. There is a possibility this is complicated by the 'auto-create table option'.

Can you share some (redacted / scrubbed) sample data for me to try to reproduce the issue?

MartinJaffer-MSFT 26,076 Reputation points

2020-08-26T21:42:59.537+00:00

@Wadhwa, Dinesh are you still facing the issue? I didn't hear back from you.
This sounds like a difficult issue. If you found a solution on your own, could you please share it here with the community?

Thank you,
Martin

Wadhwa, Dinesh 21 Reputation points

2020-09-03T06:59:42.233+00:00

@MartinJaffer-MSFT
Thanks for the reply, I was looking to this issue and according to me it is just and type mismatch between source and sink which can be fixed by going to JSON and correct it from there only instead of doing it with UI Elements. That will fix it for Sure.

There was another thing that we discussed about "Pagination is tricky to implement in Azure Data Factory"

I implemented it by looping using "Foreach activity" and calling the API by offset and PageSize which is working perfectly.

If you like the Solution, please vote and accept the answer.
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

Share via

Azure Data Factory - HTTP Connectors

0 additional answers

Your answer