Logic App Get File Content

clouddataquest 91

Hello,

i want to move big files up to 1GB with a Logic App (Consumption only), i saw there are now new Chunking methods for some Actions but not for the Get file Content, its stillt limited to about 50mb i guess.

Can you please help me? I could also use the File System connector instead von sharepoint but its limited too :( to you have any ideas? thank you very much

My Logic App:

M User's image

VenkateshDodda-MSFT 18,081 Reputation points Microsoft Employee

2023-05-15T04:05:13.31+00:00

Hello @clouddataquest ,

I hope the answer provided by @RevelinoB is helpful. Please let me know if you have any further queries.

Please accept as Yes if the answer provided is useful, so that you can help others in the community looking for remediation for similar issues.

Accepted answer

RevelinoB 2,775 Reputation points

2023-05-09T09:03:49.17+00:00

Hi Clouddataquest,

If you're trying to move large files up to 1GB using Logic Apps, there are a few options available to you.

Firstly, some actions in Logic Apps now support chunking, which allows you to split a large file into smaller pieces and process them individually. However, the "Get file content" action doesn't currently support chunking. As a workaround, you could use the "Get file metadata" action to retrieve the file size, and then use a combination of the "Get file content" and "Append to blob" actions to upload the file in chunks.

Alternatively, you could use the "FTP" connector in Logic Apps to transfer files using the FTP protocol. This approach doesn't support chunking, so you may need to manually split the file into smaller pieces before uploading.

Another option is to use Azure Blob Storage, which is a cloud-based storage service designed for storing large amounts of unstructured data. You can use the "Create blob" action in the "Azure Blob Storage" connector to upload your large files to Blob Storage, and then use the "Get blob content" action to download them again if needed. Blob Storage supports chunking, so you can upload and download files of any size.

In summary, while the "Get file content" action doesn't currently support chunking, you can work around this limitation by using other actions or connectors in Logic Apps, such as "Get file metadata," "Append to blob," "FTP," or "Azure Blob Storage."

I hope this helps! Let me know if you have any more questions.
Please sign in to rate this answer.
clouddataquest 91 Reputation points

2023-05-09T13:01:30.39+00:00

Hey Thank you for the response,

im not sure how to do this exactly.

For example I put a File (.zip file) on the Blobstorage in a folder. And can i now just use Get File Content (Blob Connector) and then the Create File (SFTP Connector)?

RevelinoB 2,775 Reputation points

2023-05-09T13:32:05.9733333+00:00

Hi Clouddatarequest,

Yes, that's correct! You can use the "Get blob content" action from the "Azure Blob Storage" connector to retrieve the contents of a file from Blob Storage, and then use the "Create file" action from the "SFTP" connector to upload the file to an SFTP server.

Here's a overview of the steps you could take:

Create a connection to your Blob Storage account using the "Azure Blob Storage" connector.

Create a connection to your SFTP server using the "SFTP" connector.

Use the "Get blob content" action to retrieve the contents of the file you want to upload from Blob Storage. You can specify the path to the file using the "Path" parameter.

Use the "Create file" action to upload the file to your SFTP server. You can specify the path to the file using the "Path" parameter.

Here's an example of what the Logic App workflow might look like:

Trigger: A file is uploaded to a specific folder in Blob Storage.

Action: Use the "Get blob content" action to retrieve the contents of the file from Blob Storage.

Action: Use the "Create file" action to upload the file to your SFTP server.

Action: (Optional) Use the "Delete blob" action to delete the original file from Blob Storage, if desired.

You may also want to consider using the "Create file" action from the "File System" connector instead of the "SFTP" connector, if your SFTP server allows you to access it using the SMB protocol. This could simplify the process and eliminate the need for an additional connection to an SFTP server.

I hope this helps! Let me know if you have any more questions.

clouddataquest 91 Reputation points

2023-05-17T13:47:19.9066667+00:00

Thank you very much! Is there maybe also a method for files bigger than 1GB ?because Blob only can transfer 1024MB

RevelinoB 2,775 Reputation points

2023-05-17T13:52:12.3366667+00:00

Hi clouddatarequest, You're welcome. Yes, you are correct that Azure Blob Storage has a limitation on individual blob sizes, with a maximum size of 1 TB (1024 GB). If you need to transfer files larger than 1 GB from Blob Storage to an SFTP server, you'll need to handle the file transfer in chunks.

Here's a high-level approach you can follow:

Use the "Get blob content" action from the "Azure Blob Storage" connector to retrieve the blob's metadata, including its size.

Check if the blob size is larger than 1 GB. If it is, proceed with chunked transfer; otherwise, you can use the regular methods for transferring files.

Calculate the number of chunks required based on the blob size and the desired chunk size. For example, if you want to transfer a 2 GB file in 500 MB chunks, you would need 4 chunks.

Iterate over the chunks and download each chunk using the "Get blob content" action with appropriate range headers to specify the portion of the blob to retrieve.

Once you have each chunk, use the "Create file" action from the "SFTP" connector to upload each chunk to the SFTP server.

On the SFTP server, you'll need to reassemble the chunks into the original file.

Please be aware that the exact implementation details might vary depending on the specific tools or programming languages you are using for integration. Make sure to consult the documentation and examples provided by the connectors and libraries you are working with for more specific guidance.

I hope this helps with your issue?

clouddataquest 91 Reputation points

2023-05-22T13:54:33.4866667+00:00

Thank you! I understand the theory but unfortunately i dont know how to make the chunking by myself i couldnt find nothing online how to do this with the apps. How to download chunks with the get blob content, where to define the range?
Sign in to comment

3 additional answers

RevelinoB 2,775 Reputation points

2023-05-22T14:00:57.08+00:00

Hi Clouddataquest,

To download the file in chunks using the 'Get blob content' action, you can define the range using the 'Range' parameter in the action. The 'Range' parameter allows you to specify the byte range of the blob content you want to retrieve.

To implement chunked downloading, you'll need to calculate the appropriate range for each chunk based on the desired chunk size and the total size of the blob. For example, if you want to download a 2 GB file in 500 MB chunks, for the first chunk you would set the 'Range' parameter to 'bytes=0-524288000' (0 to 500 MB), for the second chunk 'bytes=524288001-1048576000' (500 MB to 1 GB), and so on.

Please note that the exact syntax and format of the 'Range' parameter may vary depending on the specific tools or programming languages you are using. I would recommend referring to the documentation or examples provided by the connector or library you are using for more specific instructions on how to set the range parameter correctly.

I hope this helps with your query: "where to define the range?"
Please sign in to rate this answer.
clouddataquest 91 Reputation points

2023-05-23T06:55:32.09+00:00

Hey RevelinoB,

there is no "range" parameter in my get blob content action:

maybe because im using consumption?

RevelinoB 2,775 Reputation points

2023-05-23T07:03:48.21+00:00

Hi clouddatarequest,

Thank you for your response. If the 'Range' parameter is not available in the 'Get blob content' action you're using, it's possible that the specific implementation or library you are using does not support downloading files in chunks using byte ranges.

To confirm whether chunked downloading is supported in your case, could you please provide more details about the tools or programming language you are using for this action? That way, I can assist you better in finding an alternative solution or provide further guidance based on the specific implementation you are working with.

Additionally, if you have any documentation or examples related to the 'Get blob content' action or the library you are using, please share them. This information will be helpful in my understanding for the available options and I can provide you accurate guidance.

clouddataquest 91 Reputation points

2023-05-23T09:16:37.58+00:00

Im just using the Azure Logic App Designer i never had to choose a programming language nor library.

Maybe something with the Action Settings wrong?
M

RevelinoB 2,775 Reputation points

2023-05-23T14:45:16.9+00:00

Hi Clouddataquest,

If you're using the Azure Logic App Designer and there is no explicit option for the 'Range' parameter in the 'Get blob content' action, it's possible that the chunked downloading functionality is not directly supported within the action itself.

In this case, you can consider an alternative approach to achieve chunked downloading. One option is to use a combination of actions within your logic app to implement the chunking logic manually. Here's a high-level overview of the approach:

Use the 'Get blob properties' action to retrieve the total size of the blob.

Calculate the number of chunks based on the desired chunk size and the total size of the blob.

Set up a loop or iterative process to download each chunk sequentially.

Within the loop, use the 'Get blob content' action to download a specific chunk of the blob content.

Store or process the downloaded chunk as needed.

Repeat the loop for each subsequent chunk until all the desired chunks have been downloaded.

By manually implementing the chunking logic in this way, you can achieve the same effect as downloading the file in chunks using byte ranges.

Please note that this approach may require some custom logic and iteration within your logic app. You can use actions such as 'Control - Until', 'Control - Condition', or 'Do until' to create the loop and conditionally handle the chunking and downloading process.

I hope this helps?
Sign in to comment
Hai, Yan 0 Reputation points

2023-11-13T14:22:11.0533333+00:00
If I would like to download large file from sharepoint to Azure blob storage. can I directly use the following steps:

trigger if there is a file create/modified in sharepoint

get folder list from sharepoint

for each : azure blob storage create blob

Is there any other action I need to add?
Please sign in to rate this answer.

0 comments No comments
Sign in to comment
Mark Brown 0 Reputation points

2023-12-14T21:22:31.97+00:00

At the moment, the SharePoint connector does not have a "chunking" option for downloads. The option exists in the "create file" action but that's the only one that I've seen it on.

I had the same need and found limitations with both Power Apps and Power Automate. I ended up having to use PowerShell in a "hybrid" Azure Runbook to move the files from SharePoint to Azure Storage. As a hybrid runbook, the task actually runs on of our Azure VMs, where it has the drive space and memory that it needs. PowerShell downloads the file via MS Graph REST API and uploads it to Azure Storage, chunking any file over 1GB. The runbook is triggered by a custom form (Power App) for the SharePoint library that stores the file, which sends the item ID as a parameter. But you could also trigger it with Power Automate as well.
Please sign in to rate this answer.

0 comments No comments
Sign in to comment