How to read the file which has more than 4MB

Question

Hi All,

This is a critical requirement which needs to be completed as soon as possible, and I'm currently at a loss for solutions. The task involves converting a CSV file to base64, encrypting it, and then sending it downstream. Initially, I considered utilizing a lookup after the base64 conversion to pass the content to the PGP encryption function for encryption. However, upon observation, I noticed that after conversion, the file size exceeds 4MB, rendering the use of lookup unfeasible for obtaining the file content.

I'm seeking alternative methods to obtain the base64 content from the file. Any assistance would be greatly appreciated.

Thanks,

Jaganath

Answer 1

Glad to know that you were able to figure this out and thanks for sharing the rootcause. By the way, Microsoft Q&A community has a policy that "The question author cannot accept their own answer, they can only accept answers by others.".

Hence request you to accept this as an answer so we can close this thread. I am summarizing the issue and root cause you have found for it.

Issue Summary:

user wanted to convert a CSV file to base64, encrypting it, and then sending it downstream. Once after the conversion the file size exceeds 4MB, rendering the use of lookup unfeasible for reading the file content.

Requirement:

Seeking alternative methods to obtain the base64 content from the files.

Solution:

1. Initially, read the file and converted the content to base64 format.
2. Then, utilized the GPG function via an External Call activity, constructing a payload and passing it along.
3. Subsequently, read the response and wrote it back to a file.

Answer 2

Hi ,

Thanks for reaching out to Microsoft Q&A.

Its a limitation in adf.

The maximum supported output response payload size is 4 MB.

[https://learn.microsoft.com/en-us/answers/questions/1460361/the-length-of-execution-output-is-over-limit-(arou](https://learn.microsoft.com/en-us/answers/questions/1460361/the-length-of-execution-output-is-over-limit-(arou)

you can try to use python or powershell

import base64

def read_file_and_encode(file_path):
    with open(file_path, 'rb') as file:
        file_content = file.read()
        base64_content = base64.b64encode(file_content).decode('utf-8')
    return base64_content

# Usage
file_path = 'path/to/your/file.txt'
base64_result = read_file_and_encode(file_path)
print(base64_result)

$fileContent = Get-Content -Path 'path\to\your\file.txt' -Raw
$base64Content = [Convert]::ToBase64String([System.Text.Encoding]::UTF8.GetBytes($fileContent))
Write-Output $base64Content

Please 'Upvote'(Thumbs-up) and 'Accept' as an answer if the reply was helpful. This will benefit other community members who face the same issue.

Answer 3

Hello @Jaganath Kumar

I understand that you are looking for alternative methods to obtain the base64 content from a CSV file that exceeds 4MB after conversion.

Have you considered using the Azure Blob Storage connector You can use the "Get blob content using path" action to retrieve the content of the CSV file from Blob Storage and then pass it to the base64 conversion function. This way, you can avoid the 4MB limit of the lookup function. Once you have the base64 content, you can then pass it to the PGP encryption function for encryption. I hope this help.

---

I hope that this response has addressed your query and helped you overcome your challenges. If so, please mark this response as Answered. This will not only acknowledge our efforts, but also assist other community members who may be looking for similar solutions.