Uploading Files more than 30 MB to Azure fileshare locaton from FTP using an console application(.net-C#)

Nikhil Nair 1 Reputation point
2022-05-25T09:56:26.717+00:00

I'm using a C# (.net core) console program to upload a large file (.CSV) (limit 30 MB) to an Azure Fileshare location from an FTP site. In some cases, I'm getting incomplete writes in the Azure fileshare location. However, it works most of the time when I upload in chunks. Due to incomplete writing, the file might get damaged at the destination Fileshare location (missing a few rows and contents from the source location). The code I use is shown below. Please help me improve the procedure or provide other solutions.

public async void WriteFileToFileShare(ShareClient share, string azureFolder, string validDataFilePath, Stream stream)
        {
            ShareDirectoryClient directory = share.GetDirectoryClient(azureFolder);

            //  Azure allows for 4MB max uploads  (4 x 1024 x 1024 = 4194304)
            const int uploadLimit = 4194304;
            stream.Seek(0, SeekOrigin.Begin);   // ensure stream is at the beginning
            var fileClient = await directory.CreateFileAsync(Path.GetFileName(validDataFilePath), stream.Length);
            // If stream is below the limit upload directly
            if (stream.Length <= uploadLimit)
            {
                await fileClient.Value.UploadRangeAsync(new HttpRange(0, stream.Length), stream);
                return;
            }
            int bytesRead;
            long index = 0;
            byte[] buffer = new byte[uploadLimit];
            // Stream is larger than the limit so we need to upload in chunks
            while ((bytesRead = stream.Read(buffer, 0, buffer.Length)) > 0)
            {
                // Create a memory stream for the buffer to upload
                using MemoryStream ms = new MemoryStream(buffer, 0, bytesRead);
                await fileClient.Value.UploadRangeAsync(ShareFileRangeWriteType.Update, new HttpRange(index, ms.Length), ms);
                index += ms.Length; // increment the index to the account for bytes already written
            }
        }
Azure Files
Azure Files
An Azure service that offers file shares in the cloud.
1,156 questions
.NET CLI
.NET CLI
A cross-platform toolchain for developing, building, running, and publishing .NET applications.
322 questions
{count} votes