Only getting one element of a JSON array via API using ADF pipeline

Dave Gray 581

Hi,

Just testing out consuming REST APIs inside of ADF pipelines. I'm using a free anonymous API https://api.publicapis.org/entries which should return 1425 elements in the array entries.

User's image

I'm writing the output to an Azure SQL database table and all is fine, except I only get one row. There are no errors or feedback to indicate why just the one row.

I have not set anything specific for paging but I should not really to need to (I think) as the URL above returns everything in one page.

Can anyone please advise?

Kind regards.

Michael Taylor 51,346 Reputation points

2023-03-16T14:31:33.2733333+00:00

Please provide us the context and not just the sample data you're using. We cannot see anything about how you're calling the API, how you're adding the data to the DB or anything in between. Clearly the API is fine so the issue is in how you're doing all this and we cannot see that part.
Dave Gray 581 Reputation points

2023-03-16T15:03:00.8966667+00:00

Hi @Michael Taylor

Fair point :-)

Let me know if there's any more I need post.

Thanks

Rest dataset

Azure SQL dataset

Pipeline Source:

Source Preview

pipeline Sink

Pipeline mapping (mapped to "entries")

Settings

The output from debug gives this (there's a lot more than just this screenshot)

1 answer

ShaikMaheer-MSFT 38,406 Reputation points Microsoft Employee

2023-03-20T06:49:02.9166667+00:00

Hi Dave Gray,

Thank you for posting query in Microsoft Q&A Platform.

Your mappings tab shows its configured for only one row with index 0. Kindly use collection reference to flat your entries array as shown in below gif.

Hope this helps. Please let me know if any further questions.

Please consider hitting Accept Answer button. Accepted answers help community as well.
Please sign in to rate this answer.

2 people found this answer helpful.
ShaikMaheer-MSFT 38,406 Reputation points Microsoft Employee

2023-03-21T10:31:07.19+00:00

Hi Dave Gray, Just checking if above answer helpful? If yes, Please consider hitting Accept Answer button. Accepted answers help community as well. Please let me know if any further queries.

Dave Gray 581 Reputation points

2023-03-22T13:53:50.9233333+00:00

hi @ShaikMaheer-MSFT

Sorry, been away from work for a bit so just seen this.

I will try this out later in the week and get back to you.

Thanks for your reply BTW :-)

Imran Aslam 0 Reputation points

2023-09-14T12:35:29.9233333+00:00

Hi @ShaikMaheer-MSFT , Thank you for your detail answer on the above problem as I just faced the same issue in my ADF pipeline and got solved as per your above solutions. Thank you again. :)

Regards

Imran Aslam
Sign in to comment

Share via

Only getting one element of a JSON array via API using ADF pipeline

1 answer