Azure Data Lake Storage Gen2: BlobContainerClient.UploadBlob

Question

I am using BlobContainerClient.UploadBlob to upload files to ADLS Gen2 Storage Account. In the below scenario If we upload files from two machines with uploadblob API, it has to create directories 2023/11/12/05. When both the machines are attempting upload at the same time, will it cause any issue or UploadBlob API handles this internally? I have tried to upload from two machines, at the same time. i didn't see any issue. But would like to cross check with you. Could you please clarify here ?

Machine1 :

2023/11/12/05/1.csv

Machine2

2023/11/12/05/2.csv

Answer 1

Hi @Venkatesh Bandaru ,

Creating directories and file by upload is one (Atomic Directory Manipulation) of the many benefits of herarchical namespace. https://learn.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-namespace

One error that you can faced on future is about existing files and you can solve setting override parameter set to true.:

A RequestFailedException will be thrown if the blob already exists. To overwrite an existing block blob, get a BlobClient by calling GetBlobClient(String), and then call UploadAsync(Stream, Boolean, CancellationToken) with the override parameter set to true. https://learn.microsoft.com/en-us/dotnet/api/azure.storage.blobs.blobcontainerclient.uploadblob?view=azure-dotnet

Finally an extra information about ADLS, is about Azure blob file system has Hadoop file system logic that's why improve the operation with large amount of data.

https://learn.microsoft.com/en-us/azure/hdinsight/overview-data-lake-storage-gen2

https://www.databricks.com/glossary/hadoop-distributed-file-system-hdfs

Cheers,

Luis Arias

---

If the information helped address your question, please Accept the answer.