ADF web activity payload is to big

Question

We are using ADF now, there is a step need send Post to Remote Sever, but this payload is too big, far more than 896KB, it's already 15MB, so we try to store the request payload to blob and then use the Copy Activity, but we facing another issue is Copy Activity Source always return json payload array, but this is not accept by Remote Sever, so how can i do here to resolve this issue ?

method i tried:

1. there are no possible to change the remote server endpoint to receive json array
2. there are no way to reduce the payload cause its' file base64 string so it's big, no way to split must be in one request

have no try:

1. use azure function to upload ---- we have connection issue to remote sever
2. maybe there are other ADF price tiers we can use but i never found one show no limit for payload

Answer 1

@Ricky chen （陈帅）

Thanks for using MS Q&A platform and posting your query

I understand you're facing a challenge sending a large payload (15MB) from ADF to a remote server that doesn't accept JSON arrays. Here are some options to consider:

1. Chunking the Payload:

Even though you can't split the payload in a single request, consider chunking the data and sending it in multiple requests. This might require some logic on both the ADF and server side to handle the chunked data and reassemble it.

Here's a possible approach:

ADF: Split the payload into smaller chunks (e.g., 1MB) using functions like substring or data manipulation activities.

For each Chunk:

• Use a Web Activity to send a POST request with the chunk data and a flag indicating it's a part or the final part.
• The server should be able to identify and handle these flags to reconstruct the complete payload.

2. Azure Function as an intermediary:

If you can't modify the remote server and have connection limitations, consider using an Azure Function as an intermediary. Here's the flow:

ADF: Use a Web Activity to send the entire payload to the Azure Function.

Azure Function:

• Receive the payload from ADF.
• Split the payload into chunks if needed.
• Send individual POST requests with the chunks to the remote server.
• Handle any responses from the server and return a success/failure message to ADF.

3. Investigate ADF Pricing Tiers:

While there's no documented payload limit for ADF Web Activities, explore higher pricing tiers (e.g., Integration Runtime Managed) that might offer better performance for larger payloads. However, this might not be the most cost-effective solution for a single large payload scenario.

Hope this helps. Do let us know if you any further queries.