ADF Copy Activity. Rate limit issue with REST dataset

Question

Hi, I'm trying to copy data from a REST API to my Azure SQL database using Data Factory.

The REST API has a rate limit of 1 request per minute. In my copy activity, pagination is disabled, and the degree of copy parallelism is set to 1. However, ADF is still sending 2 requests instead of 1, which violates the rate limit policy.

When debugging, I noticed that the "Peak connections" value is 2, and I suspect this might be causing the issue, although I'm not certain. Do you know how I can limit the peak connections to 1, or do you have any other suggestions to resolve this problem?

Answer 1

Hello @Serhii Vaiser

Regarding your question about limiting the peak connections to 1, unfortunately, ADF does not provide an option to limit the peak connections. However, there are a few things you can try to resolve this issue:

1. You can try setting the "Concurrency" property to 1 in the REST dataset configuration. This will limit the number of concurrent requests to 1, which might help in your case.
2. Another option is to use the "Wait" activity in your pipeline to wait for 1 minute before sending the next request. This will ensure that you are not violating the rate limit policy.
3. You can also try using the "Lookup" activity to check the rate limit status before sending the request. If the rate limit is reached, you can wait for 1 minute before sending the next request. Regarding the "Peak connections" value of 2, this might not be causing the issue. The "Peak connections" value represents the maximum number of connections that were open at any given time during the copy activity. It does not necessarily mean that two requests were sent at the same time.

---

I hope this helps.