Uploading Files more than 30 MB to Azure fileshare locaton from FTP using an console application(.net-C#)
Nikhil Nair
1
Reputation point
I'm using a C# (.net core) console program to upload a large file (.CSV) (limit 30 MB) to an Azure Fileshare location from an FTP site. In some cases, I'm getting incomplete writes in the Azure fileshare location. However, it works most of the time when I upload in chunks. Due to incomplete writing, the file might get damaged at the destination Fileshare location (missing a few rows and contents from the source location). The code I use is shown below. Please help me improve the procedure or provide other solutions.
public async void WriteFileToFileShare(ShareClient share, string azureFolder, string validDataFilePath, Stream stream)
{
ShareDirectoryClient directory = share.GetDirectoryClient(azureFolder);
// Azure allows for 4MB max uploads (4 x 1024 x 1024 = 4194304)
const int uploadLimit = 4194304;
stream.Seek(0, SeekOrigin.Begin); // ensure stream is at the beginning
var fileClient = await directory.CreateFileAsync(Path.GetFileName(validDataFilePath), stream.Length);
// If stream is below the limit upload directly
if (stream.Length <= uploadLimit)
{
await fileClient.Value.UploadRangeAsync(new HttpRange(0, stream.Length), stream);
return;
}
int bytesRead;
long index = 0;
byte[] buffer = new byte[uploadLimit];
// Stream is larger than the limit so we need to upload in chunks
while ((bytesRead = stream.Read(buffer, 0, buffer.Length)) > 0)
{
// Create a memory stream for the buffer to upload
using MemoryStream ms = new MemoryStream(buffer, 0, bytesRead);
await fileClient.Value.UploadRangeAsync(ShareFileRangeWriteType.Update, new HttpRange(index, ms.Length), ms);
index += ms.Length; // increment the index to the account for bytes already written
}
}