The error code 2001 in ADF related to an HTTP GET activity typically occurs when the output data exceeds the current limit, which is around 4MB by default. This limit is set for the HTTP dataset and activities like Web, HTTP, or REST in Azure Data Factory. To work around this limitation, you have a few options:
If the API supports pagination, you can configure the activity to retrieve data in chunks instead of retrieving all the data in one request. This way, each request stays below the 4MB limit. To do this:
Check if the API provides pagination tokens or parameters (e.g., nextPage
, pageSize
, .
Configure your HTTP GET activity with those parameters and set the pagination rules in the Pagination
section.
If the output data is in a binary format (for example files), you can switch the activity to binary mode, which can handle larger data. You can use this in cases where the output is too large for JSON or text formats.
In the HTTP dataset settings, you can select the format as Binary.
In the activity, use the copy activity to move the binary content into Azure Blob Storage.
If you're dealing with large output, a common practice is to store the results directly into Azure Blob Storage instead of handling them in the pipeline itself.
Use a copy activity after the HTTP GET to move the large output directly into Blob Storage.
The sink of the copy activity can be Azure Blob, and the source will be the HTTP GET activity.
As of now, there is no direct configuration in Azure Data Factory to increase the output size limit for HTTP GET or similar activities. Therefore, if you're unable to paginate or switch to binary mode, splitting the request into smaller chunks or storing the output in blob storage is the best approach.