How to find the size of a folder in azure data lake Gen2

Ravi Sai Somasekhar Mahasiva 21

Hi Team,

I have a requirement to check the folder size in a container.
Eg: we have 5 folders in a container, we need to check size of each folder.

Is it possible to check size of folder in data lake Gen2 using blob fuse (Linux command), we are using command du -h --max-depth=1 (but we are getting size as 0), how to check the folder size using blob fuse. Could you please share your thoughts on it.

Thanks

PRADEEPCHEEKATLA 90,226 Reputation points

2023-02-09T04:14:58.8366667+00:00

Hello @Ravi Sai Somasekhar Mahasiva,

Just checking in to see if the below answer helped. If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.

1 answer

PRADEEPCHEEKATLA 90,226 Reputation points

2023-01-20T05:23:25.5866667+00:00

Hello @Ravi Sai Somasekhar Mahasiva,

Thanks for the question and using MS Q&A platform.

You can easily get the size of a folder in ADLS Gen2 using Azure Storage Explorer

Step1: Download the Azure Storage Explorer: https://azure.microsoft.com/en-us/products/storage/storage-explorer

Step2: Install the storage explorer and then Sign-in with your Azure Account.

Step3: Open the folder which you would like to view the size details.

On the top bar menu choose More -> Folder Statistics will help you get the details of the Directory including the size in bytes.

You can also get this details programatically using PowerShell or Python

Powershell: https://azuredude.blogspot.com/2020/10/get-azure-data-lake-gen2-container.html

Python: https://cloudarchitected.com/2019/05/computing-total-storage-size-of-a-folder-in-azure-data-lake-storage-gen2/

Hope this helps. Do let us know if you any further queries.

Please don’t forget to Accept Answer wherever the information provided helps you, this can be beneficial to other community members.
Please sign in to rate this answer.

4 people found this answer helpful.
Ravi Sai Somasekhar Mahasiva 21 Reputation points

2023-01-20T05:37:28.1533333+00:00

Hi PRADEEPCHEEKATLA-MSFT,
Thanks for the update, but I'm using blob fuse to access the container, could you please help me with the command in (Linux) to check the folder size, we are using (du -h --max-depth=1 but it is not working).

Thanks

PRADEEPCHEEKATLA 90,226 Reputation points

2023-01-24T08:25:00.4666667+00:00

Looping @SaiKishor-MSFT & @Sumarigo-MSFT Add insights on blob fuse feature?

SaiKishor-MSFT 17,236 Reputation points

2023-01-31T23:20:13.0333333+00:00

I am reaching out to the PG on this one. I will respond once I have more information. Thanks!

PRADEEPCHEEKATLA 90,226 Reputation points

2023-02-03T06:54:11.85+00:00

Hello @Ravi Sai Somasekhar Mahasiva,

Unfortunately, it is not possible via blob fuse. Calculating the size of a folder means we need to go through the list and count size of all files and subfolders within it.

As Azure storage can have millions of blobs inside a folder iterating and computing the size is costly both in terms of processing/network as well as user billing (as this requires making additional calls to storage for listing and getting properties). Hence this was by design skipped.

If user executes a “ls -l” command on a given folder the output list will also show size of each file and a small script can be written to sum it up if you intend to calculate the folder size.

Hope this helps.

PRADEEPCHEEKATLA 90,226 Reputation points

2023-02-14T10:03:30.24+00:00

Hello @Ravi Sai Somasekhar Mahasiva,

Just checking in to see if the above answer helped. If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.

Hashan Pasindu 0 Reputation points

2023-11-09T04:49:37.2+00:00

Is there a Cost involve with this Operation ?

eg: storage read/write cost, Computation cost etc.
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

Share via

How to find the size of a folder in azure data lake Gen2

1 answer

Your answer