question

SvenPeeters-3751 avatar image
0 Votes"
SvenPeeters-3751 asked SvenPeeters-3751 commented

Unable to figure out how to add data to existing data lake file

Hi,

I'm using the Azure SDK for .NET to manipulate files on the data lake (Gen2)
Within an Azure Function, I would like to add some data to a csv file stored on the data lake.

I came up with this method, should work according to the documentation (or I did not fully understand it).

Problem is that the data is not 'flushed' to the file. It remains the original content.
Can't figure out what's going on here i'm afraid :-(

Any tips ?

Regards,
Sven Peeters

PS : I must add data incrementally, otherwise the memory consumption can become an issue here.

 public void AddFileContents(string fullPath, string content, string leaseId = null)
         {
             DataLakeFileClient dataLakeFileClient = GetFileSystemClient().GetFileClient(fullPath);
             dataLakeFileClient.CreateIfNotExists();
    
             long currentLength = dataLakeFileClient.GetProperties().Value.ContentLength;
    
             byte[] byteArray = Encoding.UTF8.GetBytes(content);
             MemoryStream mStream = new MemoryStream(byteArray);
             long fileSize = mStream.Length;
    
             dataLakeFileClient.Append(mStream, currentLength, leaseId: leaseId);
             dataLakeFileClient.Flush(position: currentLength, close: true, conditions: new DataLakeRequestConditions() { LeaseId = leaseId });
         }


dotnet-csharpazure-functionsazure-data-lake-storage
· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Sven,

I'm taking a look at your issue andwill get back to you with what I find.

0 Votes 0 ·

1 Answer

SamaraSoucy-MSFT avatar image
1 Vote"
SamaraSoucy-MSFT answered SvenPeeters-3751 commented

This is very close, you should only need to make one small adjustment to the Flush() parameters on line 13. The position parameter must be equal to the length of the file after all data has been written.

I was able to get this working by adding your fileSize variable to currentLength:

 dataLakeFileClient.Flush(position: currentLength + fileSize, close: true, conditions: new DataLakeRequestConditions() { LeaseId = leaseId });


· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

I figured it out yesterday too, was planning to update my post today.

Nevertheless , Thank you very much for your clear solution

1 Vote 1 ·