Azure Pipeline Copy Rest Api to storage returning only 1 row

Vamsi Madhav 46

I have a pipeline built in azure synapse pipeline - that reads data from a rest API source and a target sink as blob storage. the pipeline was working fine with and without pagination till now. But since a while it has been returning only 1 row regardless of how the base URL is changed to.

I have tried creating a new pipeline with the same linked service
i have tried creating a new linked service
i have tried running the pipeline for a different data point
only in fail, no matter what the rows returned are only 1.

i have tested the query on postman to confirm that it does actually have 8151 rows.

weirdly i tried another pipeline with a new linkedservice for a different restapi end point and that returns 10000 (offset) rows.

so is there something that i need to do to reset the first linked service and its call ? please help

MartinJaffer-MSFT 26,086 Reputation points

2022-09-15T01:38:29.757+00:00

Hello and welcome to Microsoft Q&A @Vamsi Madhav

A Copy activity REST -> blob returning only 1 row? Hmmm... Most of the explanations that jump to mind don't seem applicable on second thought.

Have you looked at the contents of this row? What does it output? Might it be an error message, or perhaps all the rows crammed into a single row?
Has the dataset schema or copy mapping changed? Does it work if you empty both of those?

Does your api have something that lets you control how many rows are returned per call? If someone set that to 1, it would explain.

It is good you have tested with postman. It tells me the problem is most likely a difference between how postman set up and ADF set up. Might the source REST API have implemented a version change?
MartinJaffer-MSFT 26,086 Reputation points

2022-10-05T16:57:23.287+00:00

Hello @Vamsi Madhav ,
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others . Otherwise, will respond back with the more details and we will try to help .

Thanks
Martin
Vamsi Madhav 46 Reputation points

2022-10-06T10:39:41.617+00:00

Hello @MartinJaffer-MSFT ,

apologies for not getting back sooner. my first mistake was assuming implicit mapping would work for this. so that was clearly why it was just dumping it all like one big row - i think you mentioned that point about content. which didnt look right to parse the data anyways.

so for now i have gone past the learning curve hopefully and started generating the mapping dynamically to achieve the right data flow. hitting the right collection reference has so far been the key to parse accurate number of columns..

pagination was the next hurdle - but got something that works for me right now. incrementing it in 100's - could pursue any limitations on the source but for now diverted attention to something else :)

thanks for taking time to respond.
MartinJaffer-MSFT 26,086 Reputation points

2022-10-11T18:37:30.437+00:00

Thanks for the feedback!

Share via

Azure Pipeline Copy Rest Api to storage returning only 1 row

Your answer