How to scale up limit of the length of execution output(around 4MB currently)

zmsoft 60 Reputation points
2024-10-14T08:36:18.3433333+00:00

How to scale up limit of the length of execution output(around 4MB currently) ?

Error code : 2001

HTTP GET Activity

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,806 questions
0 comments No comments
{count} votes

Accepted answer
  1. Amira Bedhiafi 25,946 Reputation points
    2024-10-14T11:23:45.8833333+00:00

    The error code 2001 in ADF related to an HTTP GET activity typically occurs when the output data exceeds the current limit, which is around 4MB by default. This limit is set for the HTTP dataset and activities like Web, HTTP, or REST in Azure Data Factory. To work around this limitation, you have a few options:

    If the API supports pagination, you can configure the activity to retrieve data in chunks instead of retrieving all the data in one request. This way, each request stays below the 4MB limit. To do this:

    Check if the API provides pagination tokens or parameters (e.g., nextPage, pageSize, .

    Configure your HTTP GET activity with those parameters and set the pagination rules in the Pagination section.

    If the output data is in a binary format (for example files), you can switch the activity to binary mode, which can handle larger data. You can use this in cases where the output is too large for JSON or text formats.

    In the HTTP dataset settings, you can select the format as Binary.

    In the activity, use the copy activity to move the binary content into Azure Blob Storage.

    If you're dealing with large output, a common practice is to store the results directly into Azure Blob Storage instead of handling them in the pipeline itself.

    Use a copy activity after the HTTP GET to move the large output directly into Blob Storage.

    The sink of the copy activity can be Azure Blob, and the source will be the HTTP GET activity.

    As of now, there is no direct configuration in Azure Data Factory to increase the output size limit for HTTP GET or similar activities. Therefore, if you're unable to paginate or switch to binary mode, splitting the request into smaller chunks or storing the output in blob storage is the best approach.

    0 comments No comments

0 additional answers

Sort by: Most helpful

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.