Got 24 TB of data on an azure managed disk.

Question

Got 24 TB of data on an azure managed disk. The data is still being transferred to azure. The 32 TB datadisk has 20000 IOPS listed on Azure, however, im at roughly 19 TB and the ls -l command is sluggish. Takes about 10 seconds to complete. Should I create a blob storage and attach it instead? I read that managed disk has higher performance than blob.

Answer 1

If you are experiencing sluggish performance with your 24 TB Azure managed disk during data transfer, there are a few considerations and options you can explore:

Provisioned IOPS: While the 32 TB data disk is listed with 20,000 IOPS, it's important to note that the actual performance you experience can be affected by various factors such as workload characteristics, disk size, and the Azure VM instance type. Make sure you have provisioned the appropriate number of IOPS for your workload. You can consider increasing the provisioned IOPS for better performance.

Disk Size: The size of the disk can impact performance. As you approach the maximum capacity of the managed disk, performance can be affected. It's generally recommended to leave some free space on the disk to maintain optimal performance.

Azure Blob Storage: If you require higher performance or if the data you are transferring is primarily for archival or infrequent access purposes, you could consider using Azure Blob Storage instead. Blob Storage is suitable for scenarios where you don't require block-level access to the data and can provide cost-effective storage options. However, it's important to note that Blob Storage has different performance characteristics compared to managed disks, and it may not be suitable for all types of workloads.

Disk Caching: Consider the caching configuration for your managed disk. Azure managed disks support both read caching and write caching. Depending on your workload, enabling or adjusting the caching settings can help improve performance. However, it's important to carefully evaluate the caching requirements based on your specific workload characteristics.

Performance Monitoring: Monitor the performance metrics for your managed disk to gather insights and identify any potential bottlenecks or performance issues. Azure provides various monitoring and diagnostics tools that can help you analyze and optimize the performance of your managed disk.

Ultimately, the decision to switch to Azure Blob Storage or continue using managed disks depends on your specific requirements, workload characteristics, and performance expectations. It's recommended to evaluate the performance metrics, workload patterns, and consider the trade-offs between managed disks and Blob Storage to determine the best approach for your scenario.