Azure Blob Storage cluster size

Question

Hello,
we have a lot of small xml files of size < 64 kB each, if I compress it, I get files of size 12 kB.
The nature of these files is archive, they'll be read very rarely.
Can I set up something that I know at file system as cluster size?
The appropriate value'll be 512 B to spare a lot of space.

I do not know how physically are blobs stored ...

Answer 1

I've found at https://www.serverless360.com/blog/azure-blob-storage-vs-file-storage

"Azure Blob Storage contains three types of blobs: Block, Page and Append. A block is a single unit in a Blob. A Blob can contain many blocks but not more than 50,000 blocks per Blob. This means you can split a Blob into 50,000 blocks to upload to Azure Blobs storage. The minimum size of a block is 64KB and maximum is 100 MB. If you look at (for example .NET library), one of the objects is BlockBlob which is part of CloudBlockBlob class. This class offers you tons of things to work with a Block Blob."

Is this the correct ansver - minimum is 64 kB ?

It also seems that FileShare disk cluster size is also 64 kB and cannot be changed ...

Answer 2

@Jan Vávra Welcome to Microsoft Q&A platform, Thank you for posting your query here!

Azure Blob limits: Block blobs are optimized for uploading large amounts of data efficiently. Block blobs are composed of blocks, each of which is identified by a block ID. A block blob can include up to 50,000 blocks. Each block in a block blob can be a different size, up to the maximum size permitted for the service version in use. To create or modify a block blob, write a set of blocks via the Put Block operation and then commit the blocks to a blob with the Put Block List operation.

Blobs that are less than a certain size (determined by service version) can be uploaded in their entirety with a single write operation via Put Blob.

Storage clients default to a 128 MiB maximum single blob upload, settable in the Azure Storage client library for .NET version 11 by using the SingleBlobUploadThresholdInBytes property of the BlobRequestOptions object. When a block blob upload is larger than the value in this property, storage clients break the file into blocks. You can set the number of threads used to upload the blocks in parallel on a per-request basis using the ParallelOperationThreadCount property of the BlobRequestOptions object.

For more information: Please refer to this article: https://learn.microsoft.com/en-us/rest/api/storageservices/understanding-block-blobs--append-blobs--and-page-blobs

Azure Storage supports three types of blobs:

Block blobs store text and binary data. Block blobs are made up of blocks of data that can be managed individually. Block blobs can store up to about 190.7 TiB.
Append blobs are made up of blocks like block blobs, but are optimized for append operations. Append blobs are ideal for scenarios such as logging data from virtual machines.
Page blobs store random access files up to 8 TiB in size. Page blobs store virtual hard drive (VHD) files and serve as disks for Azure virtual machines. For more information about page blobs, see Overview of Azure page blobs

Refer to this article for Azure Files scalability and performance targets

1 Applies to read and write IOs (typically smaller IO sizes less than or equal to 64 KiB). Metadata operations, other than reads and writes, may be lower. 2 Subject to machine network limits, available bandwidth, IO sizes, queue depth, and other factors. For details see SMB Multichannel performance.

For example, if the file system cluster size is 4 KiB, the minimum file size will be 8 KiB. An Azure File Sync endpoint can scale up to the size of an Azure file share.

This article gives complete details of Azure Storage Limits: https://learn.microsoft.com/en-us/azure/azure-resource-manager/management/azure-subscription-service-limits#azure-files-and-azure-file-sync

o support the 200 TB blob size, we are increasing the maximum allowable block size from 100 MiB to 4,000 MiB and maintaining support for up to 50,000 blocks in a single blob. The 200 TB blob size is available for preview in all Azure public regions with hot, cool, archive and premium tiers. There are no billing changes.

Learn more to get started.

For more information

Block blobs let you upload large blobs efficiently. Block blobs are comprised of blocks, each of which is identified by a block ID. You create or modify a block blob by writing a set of blocks and committing them by their block IDs. Each block can be a different size, up to a maximum of 100 MB (4 MB for requests using REST versions before 2016-05-31), and a block blob can include up to 50,000 blocks. The maximum size of a block blob is therefore slightly more than 4.75 TB (100 MB X 50,000 blocks). For REST versions before 2016-05-31, the maximum size of a block blob is a little more than 195 GB (4 MB X 50,000 blocks). If you are writing a block blob that is no more than 256 MB (64 MB for requests using REST versions before 2016-05-31) in size, you can upload it in its entirety with a single write operation; see Put Blob.

Hope this helps!

Kindly let us know if the above helps or you need further assistance on this issue.

-----------------------------------------------------------------------------------------------------------------------------------------

Please do not forget to "Accept the answer” and “up-vote” wherever the information provided helps you, this can be beneficial to other community members.

Answer 3

If the end goal is to save the cost, we can also update the Blob tier to cool or achieve and files with these tier costs less

https://learn.microsoft.com/en-us/azure/storage/blobs/storage-blob-storage-tiers

----------

Please don't forget to Accept Answer and Up-vote if the response helped -- Vaibhav