Transferring a large payload to an api and reading the response.

Question

Transferring a large payload to an api and reading the response.

Hadi Chahine 0

I have a file (>150 MB) to transfer to an API (json lines). The API returns an "id" to track the processing of this file. I can't do this with a "Lookup" and a "Web activity" because "Lookup"s cannot load such a large payload. "Copy Data" doesn't work too because It doesn't output the response from the api (I need to access the "id" returned). Is there any way of getting this done purely in ADF?

QuantumCache 20,366 Reputation points Moderator

2023-10-31T06:49:52.3966667+00:00
@Hadi Chahine

Thank you for providing more details. Based on your requirements, you can use the Azure Data Factory (ADF) to transfer the file to Azure Blob Storage and then use the Web activity to call the API and get the response. Here's how you can do it:

Use the Azure Data Factory (ADF) to transfer the file to Azure Blob Storage.

Use the Web activity to call the API and get the response.

Use the Set Variable activity to store the response in a variable.

Use the Copy Data activity to copy the response from the variable to a destination of your choice.

Here's an example of how you can configure the Web activity to call the API:

Set the URL of the API in the URL field.

Set the method to PUT or POST, depending on the API requirements.

Set the body of the request to the contents of the file in Blob Storage.

Set the headers of the request, if required by the API.

Set the timeout of the request, if required by the API.

Once the Web activity is executed, the response from the API will be stored in the output of the activity. You can then use the Set Variable activity to store the response in a variable and use the Copy Data activity to copy the response to a destination of your choice.

Please note that the [Web activity](https://learn.microsoft.com/en-us/azure/data-factory/control-flow-web-activity#request-payload-schema:~:text=body-,Represents%20the%20payload%20that%20is%20sent%20to%20the%20endpoint.,-String%20(or%20expression) has a timeout , so if the API takes longer than that to respond, you may need to consider using a different approach.

REST endpoints that the web activity invokes must return a response of type JSON. The activity will timeout at 1 minute with an error if it does not receive a response from the endpoint. For endpoints that support Asynchronous Request-Reply pattern, the web activity will continue to wait without timeing out (upto 7 day) or till the endpoints signals completion of the job.

I hope this helps! Let me know if you have any further questions.
Hadi Chahine 0 Reputation points

2023-10-31T16:03:51.5266667+00:00

Hi,

Thanks for taking the time to reply.

How can't I set the body of the "Web" activity to use the file contents of a file from blob storage?

QuantumCache 20,366 Moderator

Hi Hadi,
Please try this,
In the dynamic content panel, select "Expression" and enter the following expression:

@base64(body('Get Blob Content'))

"name": "Call API",
    "type": "WebActivity",
    "dependsOn": [
        {
            "activity": "Get Blob Content",
            "dependencyConditions": [
                "Succeeded"
            ]
        }
    ],
    "inputs": [
        {
            "referenceName": "Blob Storage Dataset",
            "type": "DatasetReference"
        }
    ],
    "typeProperties": {
        "url": "https://api.example.com/data",
        "method": "POST",
        "headers": {
            "Content-Type": "application/json"
        },
        "body": {
            "value": "@base64(body('Get Blob Content'))",
            "type": "Expression"
        }
    }

Hadi Chahine 0 Reputation points

2023-10-31T17:08:22.78+00:00

Can you provide me with a full sample pipeline?

This one is giving 'body' is not a recognized function
QuantumCache 20,366 Reputation points Moderator

2023-10-31T19:50:54.8033333+00:00

Sure let me try that.

2 answers

Your answer

Hadi Chahine 0 Reputation points

2023-10-31T16:03:51.5266667+00:00

Hi,

Thanks for taking the time to reply.

How can't I set the body of the "Web" activity to use the file contents of a file from blob storage?
QuantumCache 20,366 Reputation points Moderator

2023-10-31T16:44:34.68+00:00

Hi Hadi,
Please try this,
In the dynamic content panel, select "Expression" and enter the following expression:

@base64(body('Get Blob Content'))

"name": "Call API", "type": "WebActivity", "dependsOn": [ { "activity": "Get Blob Content", "dependencyConditions": [ "Succeeded" ] } ], "inputs": [ { "referenceName": "Blob Storage Dataset", "type": "DatasetReference" } ], "typeProperties": { "url": "https://api.example.com/data", "method": "POST", "headers": { "Content-Type": "application/json" }, "body": { "value": "@base64(body('Get Blob Content'))", "type": "Expression" } }
Hadi Chahine 0 Reputation points

2023-10-31T17:08:22.78+00:00

Can you provide me with a full sample pipeline?

This one is giving 'body' is not a recognized function
QuantumCache 20,366 Reputation points Moderator

2023-10-31T19:50:54.8033333+00:00

Sure let me try that.

Answer 1

@Hadi Chahine Welcome to QnA forum.

Do you want to post the entire 150MB file to the API at once? or is it possible to post the payload to the API in Chunks?
What is the size of the response from the API?
Did you try outside of the ADF, such as Postman , traditional Web Apps to POST the Payload to the API endpoint?
What is your motivation or requirement to use only ADF in this scenario?

Web activity in Azure Data Factory and Azure Synapse Analytics

User's image

Limitations and workarounds
Here are some limitations of the Lookup activity and suggested workarounds.
User's image

Data Factory limits
Some Related info: Payload is too large
***Error message: *The payload including configurations on activity/dataSet/linked service is too large. Please check if you have settings with very large value and try to reduce its size.

Cause: The payload for each activity run includes the activity configuration, the associated dataset(s), and linked service(s) configurations if any, and a small portion of system properties generated per activity type. The limit of such payload size is 896 KB as mentioned in the Azure limits documentation for Data Factory and Azure Synapse Analytics.

Recommendation: You hit this limit likely because you pass in one or more large parameter values from either upstream activity output or external, especially if you pass actual data across activities in control flow. Check if you can reduce the size of large parameter values, or tune your pipeline logic to avoid passing such values across activities and handle it inside the activity instead.

Answer 2

Hi Satish,

Thanks for taking the time to reply to my question. The API I'm calling has to be called at once (no chunking). The response is very small (has an id). The call is being done now inside a function app, but I would like to get rid of any code outside of ADF for deployment reasons, and I want to consolidate the process within ADF as much as possible. The function app we have now gets the data from blob storage and PUTs inside the api and returns the response back to the pipeline. I'd like something that's completely within ADF.

Thanks. Let me know if that's possible. It sounds to me like a "CopyData" should do that, but the problem with "CopyData" is that it cannot return the response of the sink.

Share via

Transferring a large payload to an api and reading the response.

2 answers

Your answer