SAP HANA Azure virtual machine storage configurations
Azure provides different types of storage that are suitable for Azure VMs that are running SAP HANA. The SAP HANA certified Azure storage types that can be considered for SAP HANA deployments list like:
Azure offers two deployment methods for VHDs on Azure Standard and premium storage. We expect you to take advantage of Azure managed disk for Azure block storage deployments.
For a list of storage types and their SLAs in IOPS and storage throughput, review the Azure documentation for managed disks.
Independent of the Azure storage type chosen, the file system that is used on that storage needs to be supported by SAP for the specific operating system and DBMS. SAP support note #2972496 lists the supported file systems for different operating systems and databases, including SAP HANA. This applies to all volumes SAP HANA might access for reading and writing for whatever task. Specifically using NFS on Azure for SAP HANA, additional restrictions of NFS versions apply as stated later in this article
The minimum SAP HANA certified conditions for the different storage types are:
- Azure premium storage - /hana/log is required to be supported by Azure Write Accelerator. The /hana/data volume could be placed on premium storage without Azure Write Accelerator or on Ultra disk
- Azure Ultra disk at least for the /hana/log volume. The /hana/data volume can be placed on either premium storage without Azure Write Accelerator or in order to get faster restart times Ultra disk
- NFS v4.1 volumes on top of Azure NetApp Files for /hana/log and /hana/data. The volume of /hana/shared can use NFS v3 or NFS v4.1 protocol
Some of the storage types can be combined. For example, it's possible to put /hana/data onto premium storage and /hana/log can be placed on Ultra disk storage in order to get the required low latency. If you use a volume based on ANF for /hana/data, /hana/log volume needs to be based on NFS on top of ANF as well. Using NFS on top of ANF for one of the volumes (like /hana/data) and Azure premium storage or Ultra disk for the other volume (like /hana/log) is not supported.
In the on-premises world, you rarely had to care about the I/O subsystems and its capabilities. Reason was that the appliance vendor needed to make sure that the minimum storage requirements are met for SAP HANA. As you build the Azure infrastructure yourself, you should be aware of some of these SAP issued requirements. Some of the minimum throughput characteristics that SAP is recommending, are:
- Read/write on /hana/log of 250 MB/sec with 1 MB I/O sizes
- Read activity of at least 400 MB/sec for /hana/data for 16 MB and 64 MB I/O sizes
- Write activity of at least 250 MB/sec for /hana/data with 16 MB and 64 MB I/O sizes
Given that low storage latency is critical for DBMS systems, even as DBMS, like SAP HANA, keep data in-memory. The critical path in storage is usually around the transaction log writes of the DBMS systems. But also operations like writing savepoints or loading data in-memory after crash recovery can be critical. Therefore, it's mandatory to use Azure premium storage, Ultra disk, or ANF for /hana/data and /hana/log volumes.
Some guiding principles in selecting your storage configuration for HANA can be listed like:
- Decide on the type of storage based on Azure Storage types for SAP workload and Select a disk type
- The overall VM I/O throughput and IOPS limits in mind when sizing or deciding for a VM. Overall VM storage throughput is documented in the article Memory optimized virtual machine sizes.
- When deciding for the storage configuration, try to stay below the overall throughput of the VM with your /hana/data volume configuration. SAP HANA writing savepoints, HANA can be aggressive issuing I/Os. It's easily possible to push up to throughput limits of your /hana/data volume when writing a savepoint. If your disk(s) that build the /hana/data volume have a higher throughput than your VM allows, you could run into situations where throughput utilized by the savepoint writing is interfering with throughput demands of the redo log writes. A situation that can impact the application throughput
- If you're considering using HANA System Replication, the storage used for /hana/data on each replica must be same and the storage type used for /hana/log on each replica must be same. For example, using Azure premium storage for /hana/data with one VM and Azure Ultra disk for /hana/data in another VM running a replica of the same HANA System replication configuration, isn't supported
The suggestions for the storage configurations in this or subsequent documents are meant as directions to start with. Running workload and analyzing storage utilization patterns, you might realize that you're not utilizing all the storage bandwidth or IOPS provided. You might consider downsizing on storage then. Or in contrary, your workload might need more storage throughput than suggested with these configurations. As a result, you might need to deploy more capacity, IOPS or throughput. In the field of tension between storage capacity required, storage latency needed, storage throughput and IOPS required and least expensive configuration, Azure offers enough different storage types with different capabilities and different price points to find and adjust to the right compromise for you and your HANA workload.
Stripe sets versus SAP HANA data volume partitioning
Using Azure premium storage you may hit the best price/performance ratio when you stripe the /hana/data and/or /hana/log volume across multiple Azure disks. Instead of deploying larger disk volumes that provide the more on IOPS or throughput needed. Creating a single volume across multiple Azure disks can be accomplished with LVM and MDADM volume managers, which are part of Linux. The method of striping disks is decades old and well known. As beneficial as those striped volumes are to get to the IOPS or throughput capabilities you may need, it adds complexities around managing those striped volumes. Especially in cases when the volumes need to get extended in capacity. At least for /hana/data, SAP introduced an alternative method that achieves the same goal as striping across multiple Azure disks. Since SAP HANA 2.0 SPS03, the HANA indexserver is able to stripe its I/O activity across multiple HANA data files, which are located on different Azure disks. The advantage is that you don't have to take care of creating and managing a striped volume across different Azure disks. The SAP HANA functionality of data volume partitioning is described in detail in:
- The HANA Administrator's Guide
- Blog about SAP HANA – Partitioning Data Volumes
- SAP Note #2400005
- SAP Note #2700123
Reading through the details, it's apparent that applying this functionality takes away complexities of volume manager based stripe sets. You also realize that the HANA data volume partitioning isn't only working for Azure block storage, like Azure premium storage. You can use this functionality as well to stripe across NFS shares in case these shares have IOPS or throughput limitations.
Linux I/O Scheduler mode
Linux has several different I/O scheduling modes. Common recommendation through Linux vendors and SAP is to reconfigure the I/O scheduler mode for disk volumes from the mq-deadline or kyber mode to the noop (non-multiqueue) or none for (multiqueue) mode if not done yet by the SLES saptune profiles. Details are referenced in:
On Red Hat, leave the settings as established by the specific tune profiles for the different SAP applications.
Stripe sizes when using logical volume managers
If you're using LVM or mdadm to build stripe sets across several Azure premium disks, you need to define stripe sizes. These sizes differ between /hana/data and /hana/log. Recommendation: As stripe sizes the recommendation is to use:
- 256 KB for /hana/data
- 64 KB for /hana/log
The stripe size for /hana/data got changed from earlier recommendations calling for 64 KB or 128 KB to 256 KB based on customer experiences with more recent Linux versions. The size of 256 KB is providing slightly better performance. We also changed the recommendation for stripe sizes of /hana/log from 32 KB to 64 KB in order to get enough throughput with larger I/O sizes.
You don't need to configure any redundancy level using RAID volumes since Azure block storage keeps three images of a VHD. The usage of a stripe set with Azure premium disks is purely to configure volumes that provide sufficient IOPS and/or I/O throughput.
Accumulating multiple Azure disks underneath a stripe set, is accumulative from an IOPS and storage throughput side. So, if you put a stripe set across over 3 x P30 Azure premium storage disks, it should give you three times the IOPS and three times the storage throughput of a single Azure premium Storage P30 disk.
In case you're using LVM or mdadm as volume manager to create stripe sets across multiple Azure premium disks, the three SAP HANA FileSystems /data, /log and /shared must not be put in a default or root volume group. It's highly recommended to follow the Linux Vendors guidance which is typically to create individual Volume Groups for /data, /log and /shared.
Azure Premium Storage configurations for HANA
For detailed HANA storage configuration recommendations using Azure premium storage, read the document SAP HANA Azure virtual machine Premium SSD storage configurations
Azure Premium SSD v2 configurations for HANA
For detailed HANA storage configuration recommendations using Azure premium ssd v2 storage, read the document SAP HANA Azure virtual machine Premium SSD v2 storage configurations
Azure Ultra disk storage configuration for SAP HANA
For detailed HANA storage configuration recommendations using Azure Ultra Disk, read the document SAP HANA Azure virtual machine Ultra Disk storage configurations
NFS v4.1 volumes on Azure NetApp Files
For detail on ANF for HANA, read the document NFS v4.1 volumes on Azure NetApp Files for SAP HANA
For more information, see: