"The length of execution output is over limit (around 4MB currently)" when using "Get Metadata" activity in Azure Data Factory

Question

I want to get a list of files in a Container, Azure Blob Storage using "Get Metadata" activity in Azure Data Factory.

There are many files in the container around 10,000,000.

The activity fails getting the below error:

The length of execution output is over limit (around 4MB currently)

Is there any way that I can apply a filter in this activity to limit the output size?

Or if there is any other solution, please let me know.

Answer 1

Hello Mohammad Saber,

Thank you for posting your query here!

As we understand the ask here is if there is a way to break the 4MB limit for web activity in ADF.

Unfortunately, this is a hard limit on Web activity where the response size should be less than or equal to 4MB. This is being called out in the public documentation.

Looks like Azure Resource Manager has limits for API calls. You can make API calls at a rate within the Azure Resource Manager API limits. The maximum size of the API schema that can be used by a validation policy is 4 MB. If the schema exceeds this limit, validation policies will return errors on runtime. For more info please refer to this doc : API Management policies to validate requests and responses

And as a workaround you may try using ForEach Activity. Maybe you need to use paging query for your rest api/endpoint and return a limited number of the data each time. Then query your data in loop until the number of return data is lower than threshold value. Please refer to this source: Web activity throws overlimit error when calling rest api

Here is existing feedback on the web activity response size limitation submitted by a user, please feel free to up-vote and comment on it as it would help increase the priority of feature request suggestion. Feedback link: Azure Data Factory - Web Activity - maximize supported output response payload

Kindly let us know if you have any further queries. I’m happy to assist you further.

---

Please do not forget to "Accept the answer” and “up-vote” wherever the information provided helps you, this can be beneficial to other community members.

Answer 2

Hi,

Of course you can use filter in Get Meta Data activity. For doing, this you need to create a data set parameter for file name.

This will filter all files that start with A and end with .csv.

But, if 1000s of files fall under the same filter, again getmeta data activity cant handle >4MB. So it is a must to carefully choose the filter to not exceed 4MB limit.

thanks