Azure Blob Storage with tiered data management

Question

Hello Team,

we have the below requirements; can it be achieved by Azure Services without the need for 3rd party service:

Requirement Summary:

We want to deploy a file storage solution utilizing Azure Blob Storage with tiered data management to optimize file access, editing, and storage. 

 Required Features:

• File Access and Management:
  • Users access files through "tombstones" in cache storage, which automates the download and re-upload process to and from Azure Blob Storage. This ensures a seamless transition for file editing and storage management.
    • Files are replaced with tombstones in the cache storage when they are no longer needed, maintaining an organized and efficient storage system.
      • The system will support sharing and NTFS permissions for folders on local storages, ensuring secure and controlled access to files.
        • There will be capabilities for programmatically downloading files to local storages, allowing for automation and enhanced file management.
        • Direct Access:
          • Clients will be able to directly connect to the local NAS storage without needing an intermediary server, streamlining the access process.
            • A direct connection to the main blob storage will be enabled, necessitating that the blob storage remains unencrypted to facilitate this direct access.

Technical and Connectivity Specifications:

• Azure Blob Storage Configuration: The system includes archive, cool, and cold tiers within Azure Blob Storage to efficiently manage storage costs and access speeds.
• Local Storage/Cache Specifications:
  • Site A: Features a 1 Petabyte capacity with ~10GB/s throughput.
    • Site B: Offers a 1.6 Petabytes capacity with ~ 10GB/s throughput.
• Remote/Offsite Access
  • Remote access for the end-user to the dataset without publicly exposing the Azure Blob Container.
• Network Connectivity:
  • Site A: Equipped with a 10 Gbps express route.
    • Site B: Utilizes a 2 Gbps SD-WAN connection.

Data Management:

• File Locking: Implements a file locking mechanism when files are downloaded to cache storage and opened, preventing concurrent modifications and ensuring data integrity.

Answer 1

Hi Omar A. Ghazal - Thanks for reaching out.

The blob storage provide option to manage data via access tier and the control the lifecycle via LCM accordingly ahead.

https://learn.microsoft.com/en-us/azure/storage/blobs/access-tiers-overviewhttps://learn.microsoft.com/en-us/azure/storage/blobs/lifecycle-management-overview

From the access standpoint, Blob storage doesn't provide mechanism to host it on Windows as NFS.

Considering a 3rd party solution, it is important to first check how the customer will be accessing the storage. Is it via Access Key, SAS or if there is any option to control via Azure AD as each of these will have its own pros/cons in term of control.

https://learn.microsoft.com/en-us/azure/storage/common/authorize-data-access?toc=%2Fazure%2Fstorage%2Fblobs%2Ftoc.json

Apart from controlling on the data plane layer, you can also control on the networking plane by allowing access only via certain IP's VNET or even using PE

https://learn.microsoft.com/en-us/azure/storage/common/storage-network-security?toc=%2Fazure%2Fstorage%2Fblobs%2Ftoc.json&bc=%2Fazure%2Fstorage%2Fblobs%2Fbreadcrumb%2Ftoc.json&tabs=azure-portal

Storage follows below scalability target limits:

https://learn.microsoft.com/en-us/azure/storage/common/scalability-targets-standard-account?toc=%2Fazure%2Fstorage%2Fblobs%2Ftoc.json&bc=%2Fazure%2Fstorage%2Fblobs%2Fbreadcrumb%2Ftoc.json

For controlling the concurrency, you can make use of conditional headers or implement Lease mechanism:

https://learn.microsoft.com/en-us/azure/storage/blobs/concurrency-manage

I would recommend to review these and let us know if you have any further queries. I’m happy to assist you further.    

---

Please do not forget to "Accept the answer” and “up-vote” wherever the information provided helps you, this can be beneficial to other community members.

Answer 2

Hello Omar A. Ghazal,

Thank you for posting your query here!

File Access and Management:

Azure Blob Storage supports tiered data management with hot, cool, and archive access tiers. You can set lifecycle management policies to automatically move blobs to cooler tiers based on their age or usage patterns.

For the “tombstones” concept, Azure doesn’t provide this feature directly. However, you can implement a similar mechanism using Azure Functions or Azure Logic Apps to automate the process of downloading and re-uploading blobs.

Azure supports sharing and NTFS permissions for folders on local storages. Azure File Shares can be used with Windows’ NTFS permissions for more granular control.

For programmatically downloading files, Azure provides SDKs in various languages that you can use to interact with Blob Storage.

Direct Access:

Direct connection to the local NAS storage might not be directly supported by Azure. However, Azure File Shares can be mounted as drives on local machines.

Direct connection to the main blob storage is possible using Shared Access Signatures (SAS), which provide secure, direct access to blobs.

Technical and Connectivity Specifications:

Azure Blob Storage supports multiple access tiers including hot, cool, and archive.

Azure doesn’t directly provide local storage/cache specifications. However, Azure File Sync can be used to centralize file shares in Azure Files while maintaining the compatibility of an on-premises file server with caching capabilities.

Azure provides options for remote/offsite access to datasets in Blob Storage.

Network connectivity can be managed using Azure ExpressRoute for a private, high-speed connection between Azure and your local environment.

Data Management:

Azure Blob Storage supports leasing blobs, which provides a locking mechanism to prevent concurrent modifications.

Do let us know if you have any further queries. I’m happy to assist you further.

Please do not forget to "Accept the answer” and “up-vote” wherever the information provided helps you, this can be beneficial to other community members.

Answer 3

Thank you team for the valuable input