Edit

High availability for SAP NetWeaver on Azure VMs on Red Hat Enterprise Linux with NFS on Azure Files

  • Article

This article describes how to deploy and configure VMs, install the cluster framework, and install an HA SAP NetWeaver system, using NFS on Azure Files. The example configurations use VMs that run on Red Hat Enterprise Linux (RHEL).

Prerequisites

Overview

To deploy the SAP NetWeaver application layer, you need shared directories like /sapmnt/SID and /usr/sap/trans in the environment. Additionally, when deploying an HA SAP system, you need to protect and make highly available file systems like /sapmnt/SID and /usr/sap/SID/ASCS.

Now you can place these file systems on NFS on Azure Files. NFS on Azure Files is an HA storage solution. This solution offers synchronous Zone redundant storage (ZRS) and is suitable for SAP ASCS/ERS instances deployed across Availability Zones. You still need a Pacemaker cluster to protect single point of failure components like SAP Netweaver central services(ASCS/SCS).

The example configurations and installation commands use the following instance numbers:

Instance name Instance number
ABAP SAP Central Services (ASCS) 00
ERS 01
Primary Application Server (PAS) 02
Additional Application Server (AAS) 03
SAP system identifier NW1

SAP NetWeaver High Availability with NFS on Azure Files

Prepare infrastructure

This document assumes that you've already deployed an Azure Virtual Network, subnet and resource group.

  1. Deploy your VMs. Choose a suitable deployment type. You can deploy VMs in availability zones, if the Azure region supports zones, or in availability sets. If you need additional IP addresses for your VMs, deploy and attach a second NIC. Don’t add secondary IP addresses to the primary NIC. Azure Load Balancer Floating IP doesn't support this scenario.

  2. For your virtual IPs, deploy and configure an Azure load balancer. It's recommended to use a Standard load balancer.

    1. Configure two frontend IPs: one for ASCS (10.90.90.10) and one for ERS (10.90.90.9).
    2. Create a backend pool and add both VMs, which will be part of the cluster.
    3. Create the health probe for ASCS. The probe port is 62000. Create the probe port for ERS. The ERS probe port is 62101. When you configure the Pacemaker resources later on, you must use matching probe ports.
    4. Configure the load balancing rules for ASCS and ERS. Select the corresponding front IPs, health probes, and the backend pool. Select HA ports, increase the idle timeout to 30 minutes, and enable floating IP.

Deploy Azure Files storage account and NFS shares

NFS on Azure Files, runs on top of Azure Files Premium storage. Before setting up NFS on Azure Files, see How to create an NFS share.

There are two options for redundancy within an Azure region:

Check if your selected Azure region offers NFS 4.1 on Azure Files with the appropriate redundancy. Review the availability of Azure Files by Azure region under Premium Files Storage. If your scenario benefits from ZRS, verify that Premium File shares with ZRS are supported in your Azure region.

It's recommended to access your Azure Storage account through an Azure Private Endpoint. Make sure to deploy the Azure Files storage account endpoint and the VMs, where you need to mount the NFS shares, in the same Azure VNet or peered Azure VNets.

  1. Deploy a File Storage account named sapafsnfs. In this example, we use ZRS. If you're not familiar with the process, see Create a storage account for the Azure portal.

  2. In the Basics tab, use these settings:

    1. For Storage account name, enter sapafsnfs.
    2. For Performance, select Premium.
    3. For Premium account type, select FileStorage.
    4. For Replication, select zone redundancy (ZRS).

  3. Select Next.

  4. In the Advanced tab, deselect Require secure transfer for REST API Operations. If you don't deselect this option, you can't mount the NFS share to your VM. The mount operation will time out.

  5. Select Next.

  6. In the Networking section, configure these settings:

    1. Under Networking connectivity, for Connectivity method, select Private endpoint.
    2. Under Private endpoint, select Add private endpoint.

  7. In the Create private endpoint pane, select your Subscription, Resource group, and Location. For Name, enter sapafsnfs_pe. For Storage sub-resource, select file. Under Networking, for Virtual network, select the VNet and subnet to use. Again, you can use the VNet where your SAP VMs are, or a peered VNet. Under Private DNS integration, accept the default option Yes for Integrate with private DNS zone. Make sure to select your Private DNS Zone. Select OK.

  8. On the Networking tab again, select Next.

  9. On the Data protection tab, keep all the default settings.

  10. Select Review + create to validate your configuration.

  11. Wait for the validation to finish. Fix any issues before continuing.

  12. On the Review + create tab, select Create.

Next, deploy the NFS shares in the storage account you created. In this example, there are two NFS shares, sapnw1 and saptrans.

  1. Sign in to the Azure portal.
  2. Select or search for Storage accounts.
  3. On the Storage accounts page, select sapafsnfs.
  4. On the resource menu for sapafsnfs, select File shares under Data storage.
  5. On the File shares page, select File share.
    1. For Name, enter sapnw1, saptrans.
    2. Select an appropriate share size. For example, 128 GB. Consider the size of the data stored on the share, IOPs and throughput requirements. For more information, see Azure file share targets.
    3. Select NFS as the protocol.
    4. Select No root Squash. Otherwise, when you mount the shares on your VMs, you can't see the file owner or group.

Important

The share size above is just an example. Make sure to size your shares appropriately. Size not only based on the size of the of data stored on the share, but also based on the requirements for IOPS and throughput. For details see Azure file share targets.

The SAP file systems that don't need to be mounted via NFS can also be deployed on Azure disk storage. In this example, you can deploy /usr/sap/NW1/D02 and /usr/sap/NW1/D03 on Azure disk storage.

Important considerations for NFS on Azure Files shares

When you plan your deployment with NFS on Azure Files, consider the following important points:

  • The minimum share size is 100 GiB. You only pay for the capacity of the provisioned shares
  • Size your NFS shares not only based on capacity requirements, but also on IOPS and throughput requirements. For details see Azure file share targets
  • Test the workload to validate your sizing and ensure that it meets your performance targets. To learn how to troubleshoot performance issues with NFS on Azure Files, consult Troubleshoot Azure file shares performance
  • For SAP J2EE systems, it's not supported to place /usr/sap/<SID>/J<nr> on NFS on Azure Files.
  • If your SAP system has a heavy batch jobs load, you may have millions of job logs. If the SAP batch job logs are stored in the file system, pay special attention to the sizing of the sapmnt share. As of SAP_BASIS 7.52 the default behavior for the batch job logs is to be stored in the database. For details see Job log in the database.
  • Deploy a separate sapmnt share for each SAP system
  • Don't use the sapmnt share for any other activity, such as interfaces, or saptrans
  • Don't use the saptrans share for any other activity, such as interfaces, or sapmnt
  • Avoid consolidating the shares for too many SAP systems in a single storage account. There are also Storage account performance scale targets. Be careful to not exceed the limits for the storage account, too.
  • In general, don't consolidate the shares for more than 5 SAP systems in a single storage account. This guideline helps avoid exceeding the storage account limits and simplifies performance analysis.
  • In general, avoid mixing shares like sapmnt for non-production and production SAP systems in the same storage account.
  • We recommend deploying on RHEL 8.4 or higher to benefit from NFS client improvements.
  • Use a private endpoint. In the unlikely event of a zonal failure, your NFS sessions automatically redirect to a healthy zone. You don't have to remount the NFS shares on your VMs.
  • If you're deploying your VMs across Availability Zones, use Storage account with ZRS in the Azure regions that supports ZRS.
  • Azure Files doesn't currently support automatic cross-region replication for disaster recovery scenarios.

Setting up (A)SCS

In this example, you deploy the resources manually through the Azure portal.

Deploy Azure Load Balancer via Azure portal

After you deploy the VMs for your SAP system, create a load balancer. Then, use the VMs in the backend pool.

  1. Create an internal, standard load balancer.
    1. Create the frontend IP addresses
      1. IP address 10.90.90.10 for the ASCS
        1. Open the load balancer, select frontend IP pool, and click Add
        2. Enter the name of the new frontend IP pool (for example frontend.NW1.ASCS)
        3. Set the Assignment to Static and enter the IP address (for example 10.90.90.10)
        4. Click OK
      2. IP address 10.90.90.9 for the ASCS ERS
        • Repeat the steps above under "a" to create an IP address for the ERS (for example 10.90.90.9 and frontend.NW1.ERS)
    2. Create a single back-end pool:
      1. Open the load balancer, select Backend pools, and then select Add.
      2. Enter the name of the new back-end pool (for example, backend.NW1).
      3. Select NIC for Backend Pool Configuration.
      4. Select Add a virtual machine.
      5. Select the virtual machines of the ASCS cluster.
      6. Select Add.
      7. Select Save.
    3. Create the health probes
      1. Port 62000 for ASCS
        1. Open the load balancer, select health probes, and click Add
        2. Enter the name of the new health probe (for example health.NW1.ASCS)
        3. Select TCP as protocol, port 62000, keep Interval 5
        4. Click OK
      2. Port 62101 for ASCS ERS
        • Repeat the steps above under "c" to create a health probe for the ERS (for example 62101 and health.NW1.ERS)
    4. Load-balancing rules
      1. Create a backend pool for the ASCS
        1. Open the load balancer, select Load-balancing rules and click Add
        2. Enter the name of the new load balancer rule (for example lb.NW1.ASCS)
        3. Select the frontend IP address for ASCS, backend pool, and health probe you created earlier (for example frontend.NW1.ASCS, backend.NW1, and health.NW1.ASCS)
        4. Select HA ports
        5. Increase idle timeout to 30 minutes
        6. Make sure to enable Floating IP
        7. Click OK
        • Repeat the steps above to create load balancing rules for ERS (for example lb.NW1.ERS)

Important

Floating IP is not supported on a NIC secondary IP configuration in load-balancing scenarios. For details see Azure Load balancer Limitations. If you need additional IP address for the VM, deploy a second NIC.

Note

When VMs without public IP addresses are placed in the backend pool of internal (no public IP address) Standard Azure load balancer, there will be no outbound internet connectivity, unless additional configuration is performed to allow routing to public end points. For details on how to achieve outbound connectivity see Public endpoint connectivity for Virtual Machines using Azure Standard Load Balancer in SAP high-availability scenarios.

Important

Do not enable TCP timestamps on Azure VMs placed behind Azure Load Balancer. Enabling TCP timestamps will cause the health probes to fail. Set parameter net.ipv4.tcp_timestamps to 0. For details see Load Balancer health probes.

Create Pacemaker cluster

Follow the steps in Setting up Pacemaker on Red Hat Enterprise Linux in Azure to create a basic Pacemaker cluster for this (A)SCS server.

Prepare for SAP NetWeaver installation

The following items are prefixed with either [A] - applicable to all nodes, [1] - only applicable to node 1 or [2] - only applicable to node 2.

  1. [A] Setup host name resolution

    You can either use a DNS server or modify the /etc/hosts on all nodes. This example shows how to use the /etc/hosts file. Replace the IP address and the hostname in the following commands

    sudo vi /etc/hosts

    Insert the following lines to /etc/hosts. Change the IP address and hostname to match your environment

     # IP address of cluster node 1
 10.90.90.7    sap-cl1
 # IP address of cluster node 2
 10.90.90.8     sap-cl2
 # IP address of the load balancer frontend configuration for SAP Netweaver ASCS
 10.90.90.10   sapascs
 # IP address of the load balancer frontend configuration for SAP Netweaver ERS
 10.90.90.9    sapers

  2. [A] Install NFS client and other requirements

    sudo yum -y install nfs-utils resource-agents resource-agents-sap

  3. [1] Create the SAP directories on the NFS share.
    Mount temporarily the NFS share sapnw1 one of the VMs and create the SAP directories that will be used as nested mount points.

    # mount temporarily the volume
sudo mkdir -p /saptmp
sudo mount -t nfs sapnfs.file.core.windows.net:/sapnfsafs/sapnw1 /saptmp -o vers=4,minorversion=1,sec=sys
# create the SAP directories
sudo cd /saptmp
sudo mkdir -p sapmntNW1
sudo mkdir -p usrsapNW1ascs
sudo mkdir -p usrsapNW1ers
sudo mkdir -p usrsapNW1sys
# unmount the volume and delete the temporary directory
cd ..
sudo umount /saptmp
sudo rmdir /saptmp

  4. [A] Create the shared directories

    sudo mkdir -p /sapmnt/NW1
sudo mkdir -p /usr/sap/trans
sudo mkdir -p /usr/sap/NW1/SYS
sudo mkdir -p /usr/sap/NW1/ASCS00
sudo mkdir -p /usr/sap/NW1/ERS01

sudo chattr +i /sapmnt/NW1
sudo chattr +i /usr/sap/trans
sudo chattr +i /usr/sap/NW1/SYS
sudo chattr +i /usr/sap/NW1/ASCS00
sudo chattr +i /usr/sap/NW1/ERS01

  5. [A] Check version of resource-agents-sap

    Make sure that the version of the installed resource-agents-sap package is at least 3.9.5-124.el7

    sudo yum info resource-agents-sap

  6. [A] Add mount entries

    vi /etc/fstab
# Add the following lines to fstab, save and exit
sapnfs.file.core.windows.net:/sapnfsafs/saptrans /usr/sap/trans  nfs vers=4,minorversion=1,sec=sys  0  0
sapnfs.file.core.windows.net:/sapnfsafs/sapnw1/sapmntNW1 /sapmnt/NW1  nfs vers=4,minorversion=1,sec=sys  0  0
sapnfs.file.core.windows.net:/sapnfsafs/sapnw1/usrsapNW1sys/ /usr/sap/NW1/SYS  nfs vers=4,minorversion=1,sec=sys  0  0

# Mount the file systems
mount -a

  7. [A] Configure SWAP file

    sudo vi /etc/waagent.conf

# Set the property ResourceDisk.EnableSwap to y
# Create and use swapfile on resource disk.
ResourceDisk.EnableSwap=y

# Set the size of the SWAP file with property ResourceDisk.SwapSizeMB
# The free space of resource disk varies by virtual machine size. Make sure that you do not set a value that is too big. You can check the SWAP space with command swapon
# Size of the swapfile.
ResourceDisk.SwapSizeMB=2000

    Restart the Agent to activate the change

    sudo service waagent restart

  8. [A] RHEL configuration

    Configure RHEL as described in SAP Note 2002167 for RHEL 7.x, SAP Note 2772999 for RHEL 8.x or SAP note 3108316 for RHEL 9.x.

Installing SAP NetWeaver ASCS/ERS

  1. [1] Configure cluster default properties

    # If using RHEL 7.x
pcs resource defaults resource-stickiness=1
pcs resource defaults migration-threshold=3
# If using RHEL 8.x or later
pcs resource defaults update resource-stickiness=1
pcs resource defaults update migration-threshold=3

  2. [1] Create a virtual IP resource and health-probe for the ASCS instance

    sudo pcs node standby sap-cl2

sudo pcs resource create fs_NW1_ASCS Filesystem device='sapnfs.file.core.windows.net:/sapnfsafs/sapnw1/usrsapNW1ascs' \
  directory='/usr/sap/NW1/ASCS00' fstype='nfs' force_unmount=safe options='sec=sys,vers=4.1' \
  op start interval=0 timeout=60 op stop interval=0 timeout=120 op monitor interval=200 timeout=40 \
  --group g-NW1_ASCS

sudo pcs resource create vip_NW1_ASCS IPaddr2 \
  ip=10.90.90.10 \
  --group g-NW1_ASCS

sudo pcs resource create nc_NW1_ASCS azure-lb port=62000 \
  --group g-NW1_ASCS

    Make sure that the cluster status is ok and that all resources are started. It is not important on which node the resources are running.

    sudo pcs status

# Node sap-cl2: standby
# Online: [ sap-cl1 ]
#
# Full list of resources:
#
# rsc_st_azure    (stonith:fence_azure_arm):      Started sap-cl1
#  Resource Group: g-NW1_ASCS
#      fs_NW1_ASCS        (ocf::heartbeat:Filesystem):    Started sap-cl1 
#      nc_NW1_ASCS        (ocf::heartbeat:azure-lb):      Started sap-cl1
#      vip_NW1_ASCS       (ocf::heartbeat:IPaddr2):       Started sap-cl1

  3. [1] Install SAP NetWeaver ASCS

    Install SAP NetWeaver ASCS as root on the first node using a virtual hostname that maps to the IP address of the load balancer frontend configuration for the ASCS, for example sapascs, 10.90.90.10 and the instance number that you used for the probe of the load balancer, for example 00.

    You can use the sapinst parameter SAPINST_REMOTE_ACCESS_USER to allow a non-root user to connect to sapinst.

    # Allow access to SWPM. This rule is not permanent. If you reboot the machine, you have to run the command again.
sudo firewall-cmd --zone=public  --add-port=4237/tcp

sudo <swpm>/sapinst SAPINST_REMOTE_ACCESS_USER=sapadmin SAPINST_USE_HOSTNAME=<virtual_hostname>

    If the installation fails to create a subfolder in /usr/sap/NW1/ASCS00, try setting the owner and group of the ASCS00 folder and retry.

    sudo chown nw1adm /usr/sap/NW1/ASCS00
sudo chgrp sapsys /usr/sap/NW1/ASCS00

  4. [1] Create a virtual IP resource and health-probe for the ERS instance

    sudo pcs node unstandby sap-cl2
sudo pcs node standby sap-cl1

sudo pcs resource create fs_NW1_AERS Filesystem device='sapnfs.file.core.windows.net:/sapnfsafs/sapnw1/usrsapNW1ers' \
  directory='/usr/sap/NW1/ERS01' fstype='nfs' force_unmount=safe options='sec=sys,vers=4.1' \
  op start interval=0 timeout=60 op stop interval=0 timeout=120 op monitor interval=200 timeout=40 \
 --group g-NW1_AERS

sudo pcs resource create vip_NW1_AERS IPaddr2 \
  ip=10.90.90.9 \
 --group g-NW1_AERS

sudo pcs resource create nc_NW1_AERS azure-lb port=62101 \
 --group g-NW1_AERS

    Make sure that the cluster status is ok and that all resources are started. It is not important on which node the resources are running.

    sudo pcs status

# Node sap-cl1: standby
# Online: [ sap-cl2 ]
#
# Full list of resources:
#
# rsc_st_azure    (stonith:fence_azure_arm):      Started sap-cl2
#  Resource Group: g-NW1_ASCS
#      fs_NW1_ASCS        (ocf::heartbeat:Filesystem):    Started sap-cl2
#      nc_NW1_ASCS        (ocf::heartbeat:azure-lb):      Started sap-cl2
#      vip_NW1_ASCS       (ocf::heartbeat:IPaddr2):       Started sap-cl2
#  Resource Group: g-NW1_AERS
#      fs_NW1_AERS        (ocf::heartbeat:Filesystem):    Started sap-cl2
#      nc_NW1_AERS        (ocf::heartbeat:azure-lb):      Started sap-cl2
#      vip_NW1_AERS       (ocf::heartbeat:IPaddr2):       Started sap-cl2

  5. [2] Install SAP NetWeaver ERS

    Install SAP NetWeaver ERS as root on the second node using a virtual hostname that maps to the IP address of the load balancer frontend configuration for the ERS, for example sapers, 10.90.90.9 and the instance number that you used for the probe of the load balancer, for example 01.

    You can use the sapinst parameter SAPINST_REMOTE_ACCESS_USER to allow a non-root user to connect to sapinst.

    # Allow access to SWPM. This rule is not permanent. If you reboot the machine, you have to run the command again.
sudo firewall-cmd --zone=public  --add-port=4237/tcp

sudo <swpm>/sapinst SAPINST_REMOTE_ACCESS_USER=sapadmin SAPINST_USE_HOSTNAME=<virtual_hostname>

    If the installation fails to create a subfolder in /usr/sap/NW1/ERS01, try setting the owner and group of the ERS01 folder and retry.

    sudo chown qaadm /usr/sap/NW1/ERS01
sudo chgrp sapsys /usr/sap/NW1/ERS01

  6. [1] Adapt the ASCS/SCS and ERS instance profiles

    • ASCS/SCS profile
    sudo vi /sapmnt/NW1/profile/NW1_ASCS00_sapascs

# Change the restart command to a start command
#Restart_Program_01 = local $(_EN) pf=$(_PF)
Start_Program_01 = local $(_EN) pf=$(_PF)

# Add the keep alive parameter, if using ENSA1
enque/encni/set_so_keepalive = true

    For both ENSA1 and ENSA2, make sure that the keepalive OS parameters are set as described in SAP note 1410736.

    • ERS profile
    sudo vi /sapmnt/NW1/profile/NW1_ERS01_sapers

# Change the restart command to a start command
#Restart_Program_00 = local $(_ER) pf=$(_PFL) NR=$(SCSID)
Start_Program_00 = local $(_ER) pf=$(_PFL) NR=$(SCSID)

# remove Autostart from ERS profile
# Autostart = 1

  7. [A] Configure Keep Alive

    The communication between the SAP NetWeaver application server and the ASCS/SCS is routed through a software load balancer. The load balancer disconnects inactive connections after a configurable timeout. To prevent this, you need to set a parameter in the SAP NetWeaver ASCS/SCS profile, if using ENSA1. Change the Linux system keepalive settings on all SAP servers for both ENSA1/ENSA2. Read SAP Note 1410736 for more information.

    # Change the Linux system configuration
sudo sysctl net.ipv4.tcp_keepalive_time=300

  8. [A] Update the /usr/sap/sapservices file

    To prevent the start of the instances by the sapinit startup script, all instances managed by Pacemaker must be commented out from /usr/sap/sapservices file.

    sudo vi /usr/sap/sapservices

# On the node where you installed the ASCS, comment out the following line
# LD_LIBRARY_PATH=/usr/sap/NW1/ASCS00/exe:$LD_LIBRARY_PATH; export LD_LIBRARY_PATH; /usr/sap/NW1/ASCS00/exe/sapstartsrv pf=/usr/sap/NW1/SYS/profile/NW1_ASCS00_sapascs -D -u nw1adm

# On the node where you installed the ERS, comment out the following line
# LD_LIBRARY_PATH=/usr/sap/NW1/ERS01/exe:$LD_LIBRARY_PATH; export LD_LIBRARY_PATH; /usr/sap/NW1/ERS01/exe/sapstartsrv pf=/usr/sap/NW1/ERS01/profile/NW1_ERS01_sapers -D -u nw1adm

  9. [1] Create the SAP cluster resources.

    If using enqueue server 1 architecture (ENSA1), define the resources as follows:

    sudo pcs property set maintenance-mode=true

sudo pcs resource create rsc_sap_NW1_ASCS00 SAPInstance \
 InstanceName=NW1_ASCS00_sapascs START_PROFILE="/sapmnt/NW1/profile/NW1_ASCS00_sapascs" \
 AUTOMATIC_RECOVER=false \
 meta resource-stickiness=5000 migration-threshold=1 failure-timeout=60 \
 op monitor interval=20 on-fail=restart timeout=60 \
 op start interval=0 timeout=600 op stop interval=0 timeout=600 \
 --group g-NW1_ASCS

sudo pcs resource meta g-NW1_ASCS resource-stickiness=3000

sudo pcs resource create rsc_sap_NW1_ERS01 SAPInstance \
 InstanceName=NW1_ERS01_sapers START_PROFILE="/sapmnt/NW1/profile/NW1_ERS01_sapers" \
 AUTOMATIC_RECOVER=false IS_ERS=true \
 op monitor interval=20 on-fail=restart timeout=60 op start interval=0 timeout=600 op stop interval=0 timeout=600 \
 --group g-NW1_AERS

sudo pcs constraint colocation add g-NW1_AERS with g-NW1_ASCS -5000
sudo pcs constraint location rsc_sap_NW1_ASCS00 rule score=2000 runs_ers_NW1 eq 1
sudo pcs constraint order start g-NW1_ASCS then stop g-NW1_AERS kind=Optional symmetrical=false

sudo pcs node unstandby sap-cl1
sudo pcs property set maintenance-mode=false

    SAP introduced support for enqueue server 2, including replication, as of SAP NW 7.52. Starting with ABAP Platform 1809, enqueue server 2 is installed by default. See SAP note 2630416 for enqueue server 2 support. If using enqueue server 2 architecture (ENSA2), install resource agent resource-agents-sap-4.1.1-12.el7.x86_64 or newer and define the resources as follows:

    sudo pcs property set maintenance-mode=true

sudo pcs resource create rsc_sap_NW1_ASCS00 SAPInstance \
InstanceName=NW1_ASCS00_anftstsapvh START_PROFILE="/sapmnt/NW1/profile/NW1_ASCS00_anftstsapvh" \
AUTOMATIC_RECOVER=false \
meta resource-stickiness=5000 \
op monitor interval=20 on-fail=restart timeout=60 \
op start interval=0 timeout=600 op stop interval=0 timeout=600 \
--group g-NW1_ASCS

sudo pcs resource meta g-NW1_ASCS resource-stickiness=3000

sudo pcs resource create rsc_sap_NW1_ERS01 SAPInstance \
InstanceName=NW1_ERS01_sapers START_PROFILE="/sapmnt/NW1/profile/NW1_ERS01_sapers" \
AUTOMATIC_RECOVER=false IS_ERS=true \
op monitor interval=20 on-fail=restart timeout=60 op start interval=0 timeout=600 op stop interval=0 timeout=600 \
--group g-NW1_AERS

sudo pcs resource meta rsc_sap_NW1_ERS01  resource-stickiness=3000

sudo pcs constraint colocation add g-NW1_AERS with g-NW1_ASCS -5000
sudo pcs constraint order start g-NW1_ASCS then start g-NW1_AERS kind=Optional symmetrical=false
sudo pcs constraint order start g-NW1_ASCS then stop g-NW1_AERS kind=Optional symmetrical=false

sudo pcs node unstandby sap-cl1
sudo pcs property set maintenance-mode=false

    If you are upgrading from an older version and switching to enqueue server 2, see SAP note 2641322.

    Note

    The timeouts in the above configuration are just examples and may need to be adapted to the specific SAP setup.

    Make sure that the cluster status is ok and that all resources are started. It is not important on which node the resources are running.

    sudo pcs status

# Online: [ sap-cl1 sap-cl2 ]
#
# Full list of resources:
#
# rsc_st_azure    (stonith:fence_azure_arm):      Started sap-cl2
#  Resource Group: g-NW1_ASCS
#      fs_NW1_ASCS        (ocf::heartbeat:Filesystem):    Started sap-cl2
#      nc_NW1_ASCS        (ocf::heartbeat:azure-lb):      Started sap-cl2
#      vip_NW1_ASCS       (ocf::heartbeat:IPaddr2):       Started sap-cl2
#      rsc_sap_NW1_ASCS00 (ocf::heartbeat:SAPInstance):   Started sap-cl2
#  Resource Group: g-NW1_AERS
#      fs_NW1_AERS        (ocf::heartbeat:Filesystem):    Started sap-cl1
#      nc_NW1_AERS        (ocf::heartbeat:azure-lb):      Started sap-cl1
#      vip_NW1_AERS       (ocf::heartbeat:IPaddr2):       Started sap-cl1
#      rsc_sap_NW1_ERS01  (ocf::heartbeat:SAPInstance):   Started sap-cl1

  10. [1] Execute below step to configure priority-fencing-delay (applicable only as of pacemaker-2.0.4-6.el8 or higher)

    Note

    If you have two-node cluster, you have option to configure priority-fencing-delay cluster property. This property introduces additional delay in fencing a node that has higher total resource priority when a split-brain scenario occurs. For more information, see Can Pacemaker fence the cluster node with the fewest running resources?.

    The property priority-fencing-delay is applicable for pacemaker-2.0.4-6.el8 version or higher. If you are setting up priority-fencing-delay on existing cluster, make sure to unset pcmk_delay_max option in fencing device.

    sudo pcs resource defaults update priority=1
sudo pcs resource update rsc_sap_NW1_ASCS00 meta priority=10

sudo pcs property set priority-fencing-delay=15s

  11. [A] Add firewall rules for ASCS and ERS on both nodes.

    # Probe Port of ASCS
sudo firewall-cmd --zone=public --add-port=62000/tcp --permanent
sudo firewall-cmd --zone=public --add-port=62000/tcp
sudo firewall-cmd --zone=public --add-port=3200/tcp --permanent
sudo firewall-cmd --zone=public --add-port=3200/tcp
sudo firewall-cmd --zone=public --add-port=3600/tcp --permanent
sudo firewall-cmd --zone=public --add-port=3600/tcp
sudo firewall-cmd --zone=public --add-port=3900/tcp --permanent
sudo firewall-cmd --zone=public --add-port=3900/tcp
sudo firewall-cmd --zone=public --add-port=8100/tcp --permanent
sudo firewall-cmd --zone=public --add-port=8100/tcp
sudo firewall-cmd --zone=public --add-port=50013/tcp --permanent
sudo firewall-cmd --zone=public --add-port=50013/tcp
sudo firewall-cmd --zone=public --add-port=50014/tcp --permanent
sudo firewall-cmd --zone=public --add-port=50014/tcp
sudo firewall-cmd --zone=public --add-port=50016/tcp --permanent
sudo firewall-cmd --zone=public --add-port=50016/tcp
# Probe Port of ERS
sudo firewall-cmd --zone=public --add-port=62101/tcp --permanent
sudo firewall-cmd --zone=public --add-port=62101/tcp
sudo firewall-cmd --zone=public --add-port=3201/tcp --permanent
sudo firewall-cmd --zone=public --add-port=3201/tcp
sudo firewall-cmd --zone=public --add-port=3301/tcp --permanent
sudo firewall-cmd --zone=public --add-port=3301/tcp
sudo firewall-cmd --zone=public --add-port=50113/tcp --permanent
sudo firewall-cmd --zone=public --add-port=50113/tcp
sudo firewall-cmd --zone=public --add-port=50114/tcp --permanent
sudo firewall-cmd --zone=public --add-port=50114/tcp
sudo firewall-cmd --zone=public --add-port=50116/tcp --permanent
sudo firewall-cmd --zone=public --add-port=50116/tcp

SAP NetWeaver application server preparation

Some databases require that the database instance installation is executed on an application server. Prepare the application server virtual machines to be able to use them in these cases.

The steps below assume that you install the application server on a server different from the ASCS/SCS and HANA servers. Otherwise some of the steps below (like configuring host name resolution) are not needed.

The following items are prefixed with either [A] - applicable to both PAS and AAS, [P] - only applicable to PAS or [S] - only applicable to AAS.

  1. [A] Setup host name resolution You can either use a DNS server or modify the /etc/hosts on all nodes. This example shows how to use the /etc/hosts file. Replace the IP address and the hostname in the following commands:

    sudo vi /etc/hosts

    Insert the following lines to /etc/hosts. Change the IP address and hostname to match your environment.

    10.90.90.7    sap-cl1
10.90.90.8    sap-cl2
# IP address of the load balancer frontend configuration for SAP Netweaver ASCS
10.90.90.10   sapascs
# IP address of the load balancer frontend configuration for SAP Netweaver ERS
10.90.90.9    sapers
10.90.90.12   sapa01
10.90.90.13   sapa02

  2. [A] Create the sapmnt directory

    sudo mkdir -p /sapmnt/NW1
sudo mkdir -p /usr/sap/trans

sudo chattr +i /sapmnt/NW1
sudo chattr +i /usr/sap/trans

  3. [A] Install NFS client and other requirements

    sudo yum -y install nfs-utils uuidd

  4. [A] Add mount entries

    vi /etc/fstab
# Add the following lines to fstab, save and exit
sapnfs.file.core.windows.net:/sapnfsafs/saptrans /usr/sap/trans  nfs vers=4,minorversion=1,sec=sys  0  0
sapnfs.file.core.windows.net:/sapnfsafs/sapnw1/sapmntNW1 /sapmnt/NW1  nfs vers=4,minorversion=1,sec=sys  0  0

# Mount the file systems
mount -a

  5. [A] Configure SWAP file

    sudo vi /etc/waagent.conf

# Set the property ResourceDisk.EnableSwap to y
# Create and use swapfile on resource disk.
ResourceDisk.EnableSwap=y

# Set the size of the SWAP file with property ResourceDisk.SwapSizeMB
# The free space of resource disk varies by virtual machine size. Make sure that you do not set a value that is too big. You can check the SWAP space with command swapon
# Size of the swapfile.
ResourceDisk.SwapSizeMB=2000

    Restart the Agent to activate the change

    sudo service waagent restart

Install database

In this example, SAP NetWeaver is installed on SAP HANA. You can use every supported database for this installation. For more information on how to install SAP HANA in Azure, see High availability of SAP HANA on Azure VMs on Red Hat Enterprise Linux. For a list of supported databases, see SAP Note 1928533.

Install the SAP NetWeaver database instance as root using a virtual hostname that maps to the IP address of the load balancer frontend configuration for the database.

You can use the sapinst parameter SAPINST_REMOTE_ACCESS_USER to allow a non-root user to connect to sapinst.

# Allow access to SWPM. This rule is not permanent. If you reboot the machine, you have to run the command again.
sudo firewall-cmd --zone=public  --add-port=4237/tcp
sudo <swpm>/sapinst SAPINST_REMOTE_ACCESS_USER=sapadmin

SAP NetWeaver application server installation

Follow these steps to install an SAP application server.

  1. [A] Prepare application server

    Follow the steps in the chapter SAP NetWeaver application server preparation above to prepare the application server.

  2. [A] Install SAP NetWeaver application server.

    Install a primary or additional SAP NetWeaver applications server.

    You can use the sapinst parameter SAPINST_REMOTE_ACCESS_USER to allow a non-root user to connect to sapinst.

    sudo <swpm>/sapinst SAPINST_REMOTE_ACCESS_USER=sapadmin

  3. [A] Update SAP HANA secure store

    Update the SAP HANA secure store to point to the virtual name of the SAP HANA System Replication setup.

    Run the following command to list the entries as <sapsid>adm

    hdbuserstore List

    This should list all entries and should look similar to

    DATA FILE       : /home/nw1adm/.hdb/sapa01/SSFS_HDB.DAT
KEY FILE        : /home/nw1adm/.hdb/sapa01/SSFS_HDB.KEY

KEY DEFAULT
  ENV : 10.90.90.5:30313
  USER: SAPABAP1
  DATABASE: NW1

    In this example, the IP address of the default entry points to the VM, not the load balancer. Change the entry to point to the virtual hostname of the load balancer. Make sure to use the same port and database name. For example, 30313 and NW1 in the sample output.

    su - nw1adm
hdbuserstore SET DEFAULT nw1db:30313@NW1 SAPABAP1 <password of ABAP schema>

Test cluster setup

Thoroughly test your Pacemaker cluster. Execute the typical failover tests.

Next steps