How to fetch 100 columns fron RestAPI using Azure data Factory?

Question

I am able to fetch data from a rest API using Azure copy activity for one of the api call from LinkedIn. When I tried to make a new call to the rest API, I discovered that I could not fetch more than 20 columns at a time. There are 100 columns and I need all of then. I guess I have fetch these data in batch of 20 , which means I have to make 5 calls to the rest API. How would I achieve this if I were to use Azure data Factory for-each-loop with copy activity? The output will be csv file with all 100 columns.

Answer 1

Here is how to do a ForEach loop in Data Factory:

ForEach activity in Azure Data Factory and Azure Synapse Analytics

https://learn.microsoft.com/en-us/azure/data-factory/control-flow-for-each-activity

---

If this is helpful please accept answer.

Answer 2

Hi Pankaj Mandania ,

Thankyou for using Microsoft Q&A platform and thanks for posting your question here.

From the description of your question , it seems you are trying to fetch the entire data from REST API, however, you are only able to get 20 records. You want to know how to fetch the entire data from REST API. Please let me know if my understanding has some gap.

Copy activity supports pagination for REST APIs. You can use the pagination feature to fetch data in batches of 20 columns at a time and then use ForEach to iterate to next page and fetch next set of data.

To check out the implementation , kindly go through the below video and see if it helps:

Copy activity - Pagination rules - When API response have URL for next page

Copy data from REST API which sends response in Pages using Azure data factory

In case you have any query during implementation, feel free to share the configuration details, and error details if you may face any.

Hope it helps. Kindly accept the answer by clicking on Accept answer button. Thankyou.