How to fetch all rows from BC with skiptoken
Hey, I am using Azure data factory to pull data from my clients BC. I fetch from 19 different tables, I am using a normal copy activity for 18 of them. For the Item Ledger Entry table I am using skiptoken, the first pull I should receive all 500k entries. I find the MaxEntryNo and the next pull should retrieve entries higher than the previous MaxEntryNo. The problem I am having is it only returns the first 20k entries. When I pull again, I get 20k extra and so on. So after the first pull I have 20k entries, 2nd pull 40k entries and 3rd pull 60k entries and so on. I can copy the data without skiptoken to receive the full initial payload and then change it so it uses the skiptoken. But then the problem is, if the client has more than 20k entries a day, our consultant will only get the next 20k. How do I fetch more than 20k rows from BC using skiptoken? I've tried flirting with pagination rules to no avail.
If something is not clear or you need more information, don't hesitate to ask.
Thank you.