Error while copying Rest API to Storage in JSON

Question

Error while copying Rest API to Storage in JSON

Siva B 246

I have tried copying from Rest API into ADLS as the sink in JSON format.

and I have the below error

errorCode": "2200",
"message": "Failure happened on 'Source' side. ErrorCode=RestCallFailedWithClientError,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Rest call failed with client error, status code 429 429, please check your activity settings.\nRequest URL: https://xxxxxx.freshservice.com/api/v2/tickets?page=xxxxxx.\nResponse: ,Source=Microsoft.DataTransfer.client library,'"

It says too many requests just for the 'tickets' object and the other relative URLs are working fine while copying. Don't know what that means, Is there anything to change like retry interval or settings in Copy activity?

I'm working on a priority project and have been on this for more than a week now and no breakthrough.

Please Advice on this.

MartinJaffer-MSFT 26,236 Reputation points

2022-08-15T01:43:09.503+00:00
Hello and welcome to Microsoft Q&A @Siva B

It says too many requests just for the 'tickets' object

If the server is saying "too many requests" that sounds like a throttling issue on freshservice's side. To manage resources and ensure everyone gets a chance to use the service, often a rate-limit is imposed.

There are 3 solutions that come to mind.

some services have both single-resource requests and bulk-requests. Instead of making multiple single-resource requests rapidly, do a few bulk-requests.

make use of the activity retry options

make use of the request interval and request timeout options
Siva B 246 Reputation points

2022-08-16T22:45:50.093+00:00

For some reason, I'm not able to submit it as comment or reply, wanted to post here.

Thanks @MartinJaffer-MSFT , I have tried increasing the Request Interval and the copy activity is succeeded and the JSON file got loaded in ADLS but when I tried that JSON as the source in Dataflow it said malformed records, Parse Mode: FAILFAST. I have tried all the document forms but still says malformed records

this is how the JSON is created. It is messing with the structure of the JSON.

[{"tickets":[{"subject":"Request for Harpreet Sangha : Delete Non-starter","group_id":26000245085,"department_id":null,"category":null,"sub_category":null,"item_category":null,"requester_id":26002437095,"responder_id":null,"due_by":"2022-08-16T12:00:00Z","fr_escalated":false,"deleted":false,"spam":false,"email_config_id":null,"fwd_emails":[],"reply_cc_emails":[],"cc_emails":[],"is_escalated":false,"fr_due_by":""

When I use postman, the results are as expected, when I copy the response from the postman and saved it as a JSON file in ADLS manually and tried Dataflow, It works perfectly.

There are many issues when dealing with JSON in ADF unfortunately, All of my projects are API Integrations and JSON data and the biggest pain is that all the data is with hierarchical JSON objects and ADF doesn't support it fully and there are many issues.

Have seen many blogs and answers from Q&A on the same nested JSON in ADF and nothing is working exactly for my scenarios, I hope the ADF team will fix the limitations with JSON soon, It's strange because the data factory itself is in JSON format.

Your answer

MartinJaffer-MSFT 26,236 Reputation points

2022-08-15T01:43:09.503+00:00

Hello and welcome to Microsoft Q&A @Siva B

It says too many requests just for the 'tickets' object

If the server is saying "too many requests" that sounds like a throttling issue on freshservice's side. To manage resources and ensure everyone gets a chance to use the service, often a rate-limit is imposed.

There are 3 solutions that come to mind.

some services have both single-resource requests and bulk-requests. Instead of making multiple single-resource requests rapidly, do a few bulk-requests.

make use of the activity retry options

make use of the request interval and request timeout options
Siva B 246 Reputation points

2022-08-16T22:45:50.093+00:00

For some reason, I'm not able to submit it as comment or reply, wanted to post here.

Thanks @MartinJaffer-MSFT , I have tried increasing the Request Interval and the copy activity is succeeded and the JSON file got loaded in ADLS but when I tried that JSON as the source in Dataflow it said malformed records, Parse Mode: FAILFAST. I have tried all the document forms but still says malformed records

this is how the JSON is created. It is messing with the structure of the JSON.

[{"tickets":[{"subject":"Request for Harpreet Sangha : Delete Non-starter","group_id":26000245085,"department_id":null,"category":null,"sub_category":null,"item_category":null,"requester_id":26002437095,"responder_id":null,"due_by":"2022-08-16T12:00:00Z","fr_escalated":false,"deleted":false,"spam":false,"email_config_id":null,"fwd_emails":[],"reply_cc_emails":[],"cc_emails":[],"is_escalated":false,"fr_due_by":""

When I use postman, the results are as expected, when I copy the response from the postman and saved it as a JSON file in ADLS manually and tried Dataflow, It works perfectly.

There are many issues when dealing with JSON in ADF unfortunately, All of my projects are API Integrations and JSON data and the biggest pain is that all the data is with hierarchical JSON objects and ADF doesn't support it fully and there are many issues.

Have seen many blogs and answers from Q&A on the same nested JSON in ADF and nothing is working exactly for my scenarios, I hope the ADF team will fix the limitations with JSON soon, It's strange because the data factory itself is in JSON format.

Share via

Error while copying Rest API to Storage in JSON

Your answer