Incomplete data output from GET API ADF copy transformation

Question

I am trying to copy data using Bing GET API in ADF, during the preview data stage the data looks fine. However when pushing it to sink as JSON, the output data is incomplete. Is there any steps i missed/ misconfigured?

Configurations:

Data preview at source:

Data output in blob storage:

Answer 1

@Chun Hanssen

I'm glad that you were able to resolve your issue and thank you for posting your solution so that others experiencing the same thing can easily reference this! Since the Microsoft Q&A community has a policy that "The question author cannot accept their own answer. They can only accept answers by others ", I'll repost your solution in case you'd like to accept the answer.

Issue:

I am trying to copy data using Bing GET API in ADF, during the preview data stage the data looks fine. However when pushing it to sink as JSON, the output data is incomplete. Is there any steps i missed/ misconfigured?

Solution:

"I tested it today using the same settings for both source and sink, and it is functioning correctly now. I'm not sure if the issue was related to the API or other factors."

If I missed anything please let me know and I'd be happy to add it to my answer, or feel free to comment below with any additional information.

Hope this helps. Do let us know if you have any further queries.

---

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.

Answer 2

Hello Chun Hanssen,

Welcome to the Microsoft Q&A and thank you for posting your questions here.

I understand that you're having an incomplete data output from GET API ADF copy transformation.

Though, this is a known issue but if you do not use ADF's monitoring and logging features to capture detailed error messages and identify where the process is failing. You might need the below common way of resolving this issue.

1. If the API returns data in pages, you might only be getting the first page. Ensure that your ADF pipeline is configured to handle pagination correctly. - This is a reference to similar answer on this platform by @MartinJaffer-MSFT - https://learn.microsoft.com/en-us/answers/questions/1162055/azure-data-factory-working-with-rest-api-json-outp
2. Sometimes, the schema defined in the sink doesn't match the incoming data. Double-check your schema mappings to ensure they align with the JSON structure. You might find this answer by @ShaikMaheer-MSFT also helpful - https://learn.microsoft.com/en-us/answers/questions/722486/adf-copy-data-rest-api-putting-json-response-to-sq
3. Your data might require transformation (e.g., flattening nested JSON), ensure that you have the necessary transformations set up in your data flow. The Copy activity alone might not handle complex transformations.
4. Large datasets might be truncated if they exceed certain size limits. Check if there are any size constraints and consider breaking down the data into smaller chunks. - https://www.getorchestra.io/guides/get-from-the-adf-rest-api-change-data-capture
5. Implement error handling to catch and log any issues during the data transfer. This can help identify specific problems with the data or configuration. - https://learn.microsoft.com/en-us/answers/questions/722486/adf-copy-data-rest-api-putting-json-response-to-sq

I hope this is helpful! Do not hesitate to let me know if you have any other questions.

---

Please don't forget to close up the thread here by upvoting and accept it as an answer if it is helpful.