How to move 3GB .bak file from on-site ftp_server to azure blob storage container using sftp

August Asheim Birkeland 20

Hi,

Im trying to move a ~3GB .bak file from an ftp_server to azure blob storage using sftp. I want the transfer to happen automatically once a day, as a new .bak file with the same name is placed at the FTP_server every night.

I have looked into logical apps and their SFTP functionality, but have understood that they won't handle a file size this big?

I have therefore looked into Azure functions instead. Theese two links ([https://ourcodeworld.com/articles/read/369/how-to-access-a-sftp-server-using-ssh-net-sync-and-async-with-c-in-winforms and [https://www.c-sharpcorner.com/article/how-to-upload-files-into-azure-blog-storage-using-azure-functions-in-c-sharp/) show how you can connect to a FTP_server to fetch a file and then upload it to a blob using c# code.

My question is if using an azure function with the SSH.NET package is the goto way of solving this?

If any of you have other suggestions they are greatly appreciated!

Thanks in advance.

MughundhanRaveendran-MSFT 12,421 Reputation points

2023-01-31T10:58:47.08+00:00

@August Asheim Birkeland yes you can use a timer trigger function and use the SSH.net package to access the ftp server and download the files. Uploading the files can be done by using the Storage SDK as mentioned in this article. Please make sure that the files that is downloaded from ftp is directly stored in the blob storage as there wont be a local storage path in Azure functions.
August Asheim Birkeland 20 Reputation points

2023-02-01T13:49:43.16+00:00

@MughundhanRaveendran-MSFT thanks a lot! Could you possibly write a line of code on how i make sure the file downloaded from the FTP server is directly stored in the blob storage? eg how do i circumvent having to store the file in the azure function
MughundhanRaveendran-MSFT 12,421 Reputation points

2023-02-01T14:37:23.8866667+00:00

@August Asheim Birkeland Unfortunately there is no straight forward way to move files from ftp server to blob storage directly. Azure Functions would complicate your scenario. Also the this link that you are refering is sending the files through request body, this may work for small files but for a 3gb file, it might hit limitations. So the best way I can think of is to create a console application that can run in Azure VM. The console app can use the SSH.net package to download the files and store it in local drive and then use blob SDK to upload the files to storage container. The console app can be scheduled to run using Windows task scheduler.

Hope this helps!

August Asheim Birkeland 20

Okay thanks! @MughundhanRaveendran-MSFT So, how would you create such a console app? can i develop it on my own machine, create a executable and upload it in some way to the azure VM? and then i just have to schedule the run of the executable to once a day. Here is the code chatGPT provided me with. It doesn't tmp store the .bak file, only provides a stream so that the file is transferred directly, therefore i thought maybe i can use azure functions anyways.

using System;
using System.IO;
using System.Threading.Tasks;
using Renci.SshNet;
using Microsoft.WindowsAzure.Storage;
using Microsoft.WindowsAzure.Storage.Auth;
using Microsoft.WindowsAzure.Storage.Blob;

namespace SFTPtoBlob
{
    class Program
    {
        static void Main(string[] args)
        {
            Console.WriteLine("Fetching backup file from SFTP...");
            Task.Run(async () => await FetchFromSFTP()).Wait();
            Console.WriteLine("Done.");
        }

        private static async Task FetchFromSFTP()
        {
            var sftpClient = new SftpClient("<SFTP_endpoint>", 22, "<username>", "<password>");
            sftpClient.Connect();

            var fileStream = new MemoryStream();
            sftpClient.DownloadFile("<file_path>", fileStream);
            fileStream.Seek(0, SeekOrigin.Begin);

            Console.WriteLine("Uploading backup file to Azure Blob Storage...");
            await UploadToBlobStorage(fileStream);
            Console.WriteLine("Done.");

            sftpClient.Disconnect();
        }

        private static async Task UploadToBlobStorage(Stream fileStream)
        {
            var storageAccount = new CloudStorageAccount(
                new StorageCredentials("<account_name>", "<account_key>"), true);

            var blobClient = storageAccount.CreateCloudBlobClient();
            var container = blobClient.GetContainerReference("<container_name>");
            await container.CreateIfNotExistsAsync();

            var blob = container.GetPageBlobReference("<blob_name>");
            await blob.UploadFromStreamAsync(fileStream);
        }
    }
}

MughundhanRaveendran-MSFT 12,421 Reputation points

2023-02-02T04:04:39.37+00:00

@August Asheim Birkeland yes. Looks like this code stores the file as a stream and directly uploads to the blob container. This scenario is then feasible with the suggestion that I provided in the first comment. So you can use this code in Azure functions timer trigger.
August Asheim Birkeland 20 Reputation points

2023-02-02T13:45:11.59+00:00

@MughundhanRaveendran-MSFT okay will do. I got an error with Stream being too big. Do you reckon i have to tmp store it in between then? i showed the error and code in this question: https://learn.microsoft.com/en-us/answers/questions/1166554/max-size-of-system-io-memorystream

1 answer

JIAN WANG 335 Reputation points

2023-01-28T13:07:41.9366667+00:00
if you only wan to sync the file , i would suggest avoiding coding unless you have other need that has to be done by coding.

Use power automate.

Use AzCopy

Use onedrive

i would like to combine power automate + onedrive.
Please sign in to rate this answer.
August Asheim Birkeland 20 Reputation points

2023-02-01T13:26:06.3666667+00:00

Thanks for your answer Telen. But dont i need a temporary storage for azcopy to work? this is for a client so i don't think i can introduce OneDrive in this scenario unfortuneately

ChatGPT generated this code for me

# Download the backup file from the FTP endpoint azcopy cp "ftp://<username>:<password>@<ftp_endpoint>/<backup_file>.bak" <temp_folder> # Generate a SAS token for the destination blob storage sas_token=$(az storage blob generate-sas --account-name <account_name> --account-key <account_key> --container-name <container_name> --name <backup_file>.bak --permissions w --expiry $(date -u -d "30 minutes" '+%Y-%m-%dT%H:%M:%SZ')) # Upload the backup file to the blob storage using the SAS token azcopy cp <temp_folder>/<backup_file>.bak "https://<account_name>.blob.core.windows.net/<container_name>/<backup_file>.bak?$sas_token"

August Asheim Birkeland 20 Reputation points

2023-02-01T13:59:17.48+00:00

@JIAN WANG

JIAN WANG 335 Reputation points

2023-02-01T19:04:52.02+00:00

hi,

you do need a temp folder , since you have to install Azcopy on a computer, if you think this is the best solution for your client.

if you wan to use logic app.

use Powerautomat

for your reference

SFTP-SSH

August Asheim Birkeland 20 Reputation points

2023-02-02T14:12:45.5033333+00:00

@JIAN WANG I have done mutliple attempts with logical aps and SFTP - SSH before but it does not support file sizes of 3GB! I think its treshold is at 1GB.
Sign in to comment