Logic App failed to extract API data files exceeding 100 MB

Question

Logic App failed to extract API data files exceeding 100 MB

AMJ 6

Logic App failed to extract API data files exceeding 100 MB as below:

As you could see a data file over 100 MB has reached the limit set up by Logic App. Followed MS instruction https://learn.microsoft.com/en-us/azure/logic-apps/logic-apps-handle-large-messages as below:

It didn’t work. This is an API request worked well to extract data (< 100 MB) from a client and saved to our data lake.

I also tried to use Copy activity in data factory on the same API request to pull the data across. It worked for data files > 100 MB, but it was not sustained. When working on a number of files the transaction was interrupted often by saying the source file was not ready to be caught and sink to destination. Though I increased the timeout as below:

It didn’t help. As a matter of fact there was no timeout required to get the response with required data when working in Logic App.

In a nutshell (for API request):
• Data transaction worked well in Logic App so long as the data file is < 100 MB;
• Data transaction was intermittently interrupted in Data Factory, but the data file can be > 100 MB.

2 answers

Your answer

Answer 1

MayankBargali-MSFT 70,941 Moderator

@AMJ Apology for the delay. It looks like there are two issues. One while using the logic app HTTP connector and the other data factory activity call. I have added the azure-data-factory tag so the expert can comment on your second issue but it is always suggested to create two different issues so it would be easier for the community to search for a different issue with a different service. I will reach out to azure data factory team to comment on your second issue.

On your first issue with logic app HTTP connector. This is the hard limit of the consumption logic app as mentioned here for the Maximum input or output size and holds true for the HTTP trigger. Depending on the different connector this can go up to 1GB.

This Maximum input or output size can be changed in the case of single tenant logic app as mentioned here. With regards to the maximum limit for the single tenant logic app, it depends on various factors like what is the compute and memory size (Workflow Standard plan type) you are using and how many of such messages will be processed in parallel, etc.

AMJ 6 Reputation points

2022-02-01T06:14:16.31+00:00

@MayankBargali-MSFT Thank you very much for your prompt reply, appreciated.

As you mentioned that this Maximum input or output size can be changed in the case of single tenant logic app as below:

The question is where you would go to set the parameter "Runtime.ContentLink.MaximumContentSizeInBytes" as indicated from above. Here is the settings in my logic app as below:

As you could see there is no place for setting up the parameter "Runtime.ContentLink.MaximumContentSizeInBytes". The "High throughput" was turned on before when the logic app run into the limit of 100 MB. Would APIM be a practical approach to solve the problem from your experience?
MayankBargali-MSFT 70,941 Reputation points Moderator

2022-02-01T12:28:10.977+00:00

@AMJ As per the screenshot it looks like the logic app is under the Consumption SKU and not the Standard SKU (single tenant environment).
To create a Standard logic app please select Type as Standard under Instance details.

These settings are only available for single tenant logic app as mentioned in this document. The document covers how you can set the Runtime.Backend.HttpOperation.MaxContentSize and Runtime.FlowRunActionJob.MaximumActionResultSize when you are deploying from VS Code or you can update the same from azure portal by navigating to logic app website (i.e. scm website as logic app standard runs under function environment). The same is also documented here.

Navigate to https://<yourlogicappname>.scm.azurewebsites.net/DebugConsole
Go the path and update your host.json file : site > wwwroot > host.json

Updated Host.json file:

Successful execution of HTTP Action with 200 MB file:
MayankBargali-MSFT 70,941 Reputation points Moderator

2022-02-01T12:49:45.37+00:00

@AMJ You can either move to the logic app Standard so you can have control over these limits. But in case if you are using APIM then you may write the policy to send the data in chunks (not more than 100 MB) but still how you will be creating the file at the logic app with the chuck data send from APIM if you want to perform some logic. You need to handle it through some external storage rather than an HTTP call.

It is recommended to use blob storage to manage large uploads and the blob connector would have a higher limit of 1 GB when chucking is enabled and the same you can use in your consumption logic app.

The alternative is using the Azure function but they do have the 100 MB limit as mentioned here which you have the control using custom container approach because you could have your docker file and modify the web.config and use the rest of the functions base image as it is. But every time there is the image of a new function, you just rebuilds the container and deploys.
AMJ 6 Reputation points

2022-02-08T02:02:45+00:00

@MayankBargali-MSFT I am really sorry for the delay to get you back in time. I am the only guy working in Azure for the whole company exceeding thousands staff with multiple projects in my plates at the same time. Thank you so much for your considerate response on the matter. I am so amazed for your extensive knowledge in Azure.

Having adopted the Standard SKU as you suggested it seemed everything fits together well until the time I tried to save the API data file to the data lake (V2) with existing storage account. In my previous Consumption model I applied "Create Blob" action in the logic app to save the API data file to the designated folder path in the data lake as below:

It worked perfectly except the data file must be less than 100 MB. In the Standard approach "Create Blob" action seemed disappeared as below:

In came with "Call an Azure API Management API" when I moved actions to a higher level up as below:

Comparing the Consumption Azure blob action below:

Is it true that I need to apply the action "Call an Azure API Management API" in the Standard approach to save an API data file like the one I used "Create Blob" in the Consumption approach? If so it seemed a daunting task as you mentioned in the previous response. Is there other solutions (with less fuss) to perform such a relative simple task to save a data file to the data lake?
MayankBargali-MSFT 70,941 Reputation points Moderator

2022-02-09T06:56:12.917+00:00

@AMJ As per the 3rd screenshot on your comment it looks like either you have created the Stateless workflow (that only supports Built-in connector) or in the case of Stateflow workflow you have not selected the Azure option. Only the Azure blob connector will have the Create Blob action as shared in your 6th screenshot. You don't have to use Call an Azure API Management API action as you can use the Stateflow workflow with Azure option to list the connector. You can refer to the comparison section for more details on Stateless and Stateful.
AMJ 6 Reputation points

2022-02-10T11:34:04.63+00:00

@MayankBargali-MSFT You are right again that I could implement create blob (V2) action now by following your instruction of using the Stateflow workflow with Azure option. I am glad that HTTP action has passed 100 MB barrier, but "Create blob (V2)" failed 100 MB as illustrated below:

Any thoughts on further modifying the file host.json to overcome this hurdle?
MayankBargali-MSFT 70,941 Reputation points Moderator

2022-02-14T10:05:11.227+00:00

@AMJ You can try enabling the chunking on the Create Blob action.

The Runtime.FlowRunActionJob.MaximumActionResultSize should help with the larger size.
AMJ 6 Reputation points

2022-02-17T07:53:09.387+00:00

@MayankBargali-MSFT I realized that replacing Azure action “Create blob (V2)” by a Built-in action “Upload a Blob to Azure Storage” worked well in breaking through the 100 MB barrier to save API data files to a data lake. Even so it still froze up for a file of size 500 MB when having changed “Runtime.FlowRunActionJob.MaximumActionResultSize” to 1 GB. It seemed by increasing the size for the parameter “Runtime.FlowRunActionJob.MaximumActionResultSize” is of little practical value in the real business environment situation.

As such I decided to ditch the Logic App approach on the issue, instead, to approach Copy Data in data factory which succeeded so far for all the imported files including the one with 500 MB. By changing around the attributes “Retry” and “Retry interval (set)” etc in the Copy Data activity seemed have increased the stability of API data file transactions.

Thank you so much for your assistance and inspiration on the matter. Without your inspiration I couldn’t have gone such a long distance on the journey in building up such a profound knowledge.

Answer 2

AMJ 6

@MayankBargali-MSFT It has been turned on as below:

The Runtime.FlowRunActionJob.MaximumActionResultSize only helped with HTTP action to extract data file exceeding 100 MB, but won't help Create blob (V2) action to save a file exceeding 100 MB in the data lake as below:

I have run 2 files this time, 1 MB (succeeded) and 180 MB (failed) ad below:

The Host.json used as below:

Any more thoughts?

Share via

Logic App failed to extract API data files exceeding 100 MB

2 answers

Your answer