Error "The payload including configurations on activity/dataSet/linked service is too large. Please check if you have settings with very large value and try to reduce its size." while executing azure function in adf.

Amar Agnihotri 921

Hi,
I have an adf pipeline which calls the azure function to convert the string output to a valid json format. My pipeline works fine most of the time but some times i face this error while executing azure function activity

This is my pipeline

Can anybody tell any solution for this ?

Thanks

2 answers

Suba Balaji 11,206 Reputation points

2022-08-16T11:15:00.823+00:00

Hi @Amar Agnihotri ,

Is the string that you are forming differ in size in every run? Like for clean runs, the size is lesser and sometimes it becomes a bigger payload with huge data?

As per this documentation, the payload is limited to 896 KB.

As a workaround, you can write the string to a table column or a blob file an access that from your azure function.

Let us know
Please sign in to rate this answer.
Amar Agnihotri 921 Reputation points

2022-08-17T09:30:28.743+00:00

hi @Suba Balaji
Thanks for the reply. Actually the string being passed to azure function activity is of different size on each run but it is running successfully for the bigger strings and sometimes it gets fail for the smaller string. So, as per MS documentation if the issue is due to payload size then it shouldn't run successfully for the bigger strings. There is no issue with my azure function code also because when i checked the function in postman for the same input it was running fine.

Suba Balaji 11,206 Reputation points

2022-08-25T15:24:42.167+00:00

Hi @Amar Agnihotri

Sorry for missing this thread. Do you still face this issue or now it's ok? Please let us know

Thanks

Łukasz Lubina 1 Reputation point

2022-09-05T12:42:01.203+00:00

Hello,

Currently, I'm facing the same issue - I was taking BLOB file output via API and passing the output to another rest API call as a body (XML) to post the data.
It was working fine for 2-4MB files. Today I've done the same for the 800kb file, which gave me the error.

Amar Agnihotri 921 Reputation points

2022-09-05T12:55:04.07+00:00

@Suba Balaji , @Łukasz Lubina

Hi !
I raised a ticket in the MS support and asked them why the issue is happening intermittently while sometime the pipeline is running fine even in the case of larger payload. They told me that after digging it deeper that this issue has been considered as a bug by the product team and the expected ETA is 3-4 months. Actually, sometimes the compression technique used by adf in the back end works fine and sometime it is not able to compress and only at that time i am having this issue. The compression technique is working fine sometime for the larger payload and when it fails even for the smaller payload it shows payload error. The product team is working on the solution and will come up with the increased size of payload which would be greater than 896 kb. We can't do anything rather waiting for the ETA for now. Here is the detail of support request -
Sign in to comment
11-4688 131 Reputation points

2023-01-23T08:15:26.3033333+00:00

Any update on that? It still causes issues.
Please sign in to rate this answer.

0 comments No comments
Sign in to comment

Share via

Error "The payload including configurations on activity/dataSet/linked service is too large. Please check if you have settings with very large value and try to reduce its size." while executing azure function in adf.

2 answers