How to fetch all rows from BC with skiptoken

Valur Marvin Pálsson 1 Reputation point
2023-03-02T12:40:43.2333333+00:00

Hey, I am using Azure data factory to pull data from my clients BC. I fetch from 19 different tables, I am using a normal copy activity for 18 of them. For the Item Ledger Entry table I am using skiptoken, the first pull I should receive all 500k entries. I find the MaxEntryNo and the next pull should retrieve entries higher than the previous MaxEntryNo. The problem I am having is it only returns the first 20k entries. When I pull again, I get 20k extra and so on. So after the first pull I have 20k entries, 2nd pull 40k entries and 3rd pull 60k entries and so on. I can copy the data without skiptoken to receive the full initial payload and then change it so it uses the skiptoken. But then the problem is, if the client has more than 20k entries a day, our consultant will only get the next 20k. How do I fetch more than 20k rows from BC using skiptoken? I've tried flirting with pagination rules to no avail.

If something is not clear or you need more information, don't hesitate to ask.

Thank you.

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,625 questions
{count} votes

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.