Troubleshoot the Azure Application Consistent Snapshot (AzAcSnap) tool

Article
06/01/2023

This article describes how to troubleshoot issues when using the Azure Application Consistent Snapshot (AzAcSnap) tool for Azure NetApp Files and Azure Large Instance.

You might encounter several common issues when running AzAcSnap commands. Follow the instructions to troubleshoot the issues. If you still have issues, open a Service Request for Microsoft Support from the Azure portal and assign the request to the SAP HANA Large Instance queue.

AzAcSnap command won't run

In some cases AzAcSnap won't start due to the user's environment.

Failed to create CoreCLR

AzAcSnap is written in .NET and the CoreCLR is an execution engine for .NET apps, performing functions such as IL byte code loading, compilation to machine code and garbage collection. In this case there is an environmental problem blocking the CoreCLR engine from starting.

A common cause is limited permissions or environmental setup for the AzAcSnap operating system user, usually 'azacsnap'.

The error Failed to create CoreCLR, HRESULT: 0x80004005 can be caused by lack of write access for the azacsnap user to the system's TMPDIR.

Note

All command lines starting with # are commands run as root, all command lines starting with > are run as azacsnap user.

Check the /tmp ownership and permissions (note in this example only the root user can read and write to /tmp):

# ls -ld /tmp
drwx------ 9 root root 8192 Mar 31 10:50 /tmp

A typical /tmp has the following permissions, which would allow the azacsnap user to run the azacsnap command:

# ls -ld /tmp
drwxrwxrwt 9 root root 8192 Mar 31 10:51 /tmp

If it's not possible to change the /tmp directory permissions, then create a user specific TMPDIR.

Make a TMPDIR for the azacsnap user:

> mkdir /home/azacsnap/_tmp
> export TMPDIR=/home/azacsnap/_tmp
> azacsnap -c about

 
 
                            WKO0XXXXXXXXXXXNW
                           Wk,.,oxxxxxxxxxxx0W
                           0;.'.;dxxxxxxxxxxxKW
                          Xl'''.'cdxxxxxxxxxdkX
                         Wx,''''.,lxxxxdxdddddON
                         0:''''''.;oxdddddddddxKW
                        Xl''''''''':dddddddddddkX
                       Wx,''''''''':ddddddddddddON
                       O:''''''''',xKxddddddoddod0W
                      Xl''''''''''oNW0dooooooooooxX
                     Wx,,,,,,'','c0WWNkoooooooooookN
                    WO:',,,,,,,,;cxxxxooooooooooooo0W
                    Xl,,,,,,,;;;;;;;;;;:llooooooooldX
                   Nx,,,,,,,,,,:c;;;;;;;;coooollllllkN
                  WO:,,,,,,,,,;kXkl:;;;;,;lolllllllloOW
                  Xl,,,,,,,,,,dN WNOl:;;;;:lllllllllldK
                  0c,;;;;,,,;lK     NOo:;;:clllllllllo0W
                  WK000000000N        NK000KKKKKKKKKKXW
 
 
                Azure Application Consistent Snapshot Tool
                       AzAcSnap 7a (Build: 1AA8343)

Important

Changing the user's TMPDIR would need to be made permanent by changing the user's profile (e.g. $HOME/.bashrc or $HOME/.bash_profile). There would also be a need to clean-up the TMPDIR on system reboot, this is typically automatic for /tmp.

Check log files, result files, and syslog

Some of the best sources of information for investigating AzAcSnap issues are the log files, result files, and the system log.

Log files

The AzAcSnap log files are stored in the directory configured by the logPath parameter in the AzAcSnap configuration file. The default configuration filename is azacsnap.json, and the default value for logPath is ./logs, which means the log files are written into the ./logs directory relative to where the azacsnap command runs. If you make the logPath an absolute location, such as /home/azacsnap/logs, azacsnap always outputs the logs into /home/azacsnap/logs, regardless of where you run the azacsnap command.

The log filename is based on the application name, azacsnap, the command run with -c, such as backup, test, or details, and the default configuration filename, such as azacsnap.json. With the -c backup command, a default log filename would be azacsnap-backup-azacsnap.log, written into the directory configured by logPath.

This naming convention allows for multiple configuration files, one per database, to help locate the associated log files. If the configuration filename is SID.json, then the log filename when using the azacsnap -c backup --configfile SID.json option is azacsnap-backup-SID.log.

Result files and syslog

For the -c backup command, AzAcSnap writes to a *.result file. The purpose of the *.result file is to provide high-level confirmation of success/failure. If the *.result file is empty, then assume failure. Any output written to the *.result file is also output to the system log (for example, /var/log/messages) by using the logger command. The *.result filename has the same base name as the log file to allow for matching the result file with the configuration file and the backup log file. The *.result file goes into the same location as the other log files and is a simple one line output file.

Example for successful completion:

Output to *.result file:
```
Database # 1 (PR1) : completed ok
```

Output to /var/log/messages:

Dec 17 09:01:13 azacsnap-rhel azacsnap: Database # 1 (PR1) : completed ok

Example output where a failure has occurred and AzAcSnap captured the failure:
1. Output to *.result file:
```
Database # 1 (PR1) : failed
```
2. Output to /var/log/messages:
```
Dec 19 09:00:30 azacsnap-rhel azacsnap: Database # 1 (PR1) : failed
```

Troubleshoot failed 'test storage' command

The command azacsnap -c test --test storage might not complete successfully.

Check network firewalls

Communication with Azure NetApp Files might fail or time out. To troubleshoot, make sure firewall rules aren't blocking outbound traffic from the system running AzAcSnap to the following addresses and TCP/IP ports:

https://management.azure.com:443
https://login.microsoftonline.com:443

Use Cloud Shell to validate configuration files

You can test whether the service principal is configured correctly by using Cloud Shell through the Azure portal. Using Cloud Shell tests for correct configuration, bypassing network controls within a virtual network or virtual machine (VM).

In the Azure portal, open a Cloud Shell session.
Make a test directory, for example mkdir azacsnap.
Switch to the azacsnap directory, and download the latest version of AzAcSnap.
```
wget https://aka.ms/azacsnapinstaller
```
Make the installer executable, for example chmod +x azacsnapinstaller.

Extract the binary for testing.

./azacsnapinstaller -X -d .

The results look like the following output:

+-----------------------------------------------------------+
| Azure Application Consistent Snapshot Tool Installer |
+-----------------------------------------------------------+
|-> Installer version '5.0.2_Build_20210827.19086'
|-> Extracting commands into ..
|-> Cleaning up .NET extract dir

Use the Cloud Shell Upload/Download icon to upload the service principal file, azureauth.json, and the AzAcSnap configuration file, such as azacsnap.json, for testing.
Run the storage test.
```
./azacsnap -c test --test storage
```
Note

The test command can take about 90 seconds to complete.

Failed test on Azure Large Instance

The following error example is from running azacsnap on Azure Large Instance:

azacsnap -c test --test storage

The authenticity of host '172.18.18.11 (172.18.18.11)' can't be established.
ECDSA key fingerprint is SHA256:QxamHRn3ZKbJAKnEimQpVVCknDSO9uB4c9Qd8komDec.
Are you sure you want to continue connecting (yes/no)?

To troubleshoot this error, don't respond yes. Make sure that your storage IP address is correct. You can confirm the storage IP address with the Microsoft operations team.

The error usually appears when the Azure Large Instance storage user doesn't have access to the underlying storage. To determine whether the storage user has access to storage, run the ssh command to validate communication with the storage platform.

ssh <StorageBackupname>@<Storage IP address> "volume show -fields volume"

The following example shows the expected output:

ssh clt1h80backup@10.8.0.16 "volume show -fields volume"

vserver volume
--------------------------------- ------------------------------
osa33-hana-c01v250-client25-nprod hana_data_h80_mnt00001_t020_vol
osa33-hana-c01v250-client25-nprod hana_data_h80_mnt00002_t020_vol

Failed test with Azure NetApp Files

The following error example is from running azacsnap with Azure NetApp Files:

azacsnap --configfile azacsnap.json.NOT-WORKING -c test --test storage

BEGIN : Test process started for 'storage'
BEGIN : Storage test snapshots on 'data' volumes
BEGIN : 1 task(s) to Test Snapshots for Storage Volume Type 'data'
ERROR: Could not create StorageANF object [authFile = 'azureauth.json']

To troubleshoot this error:

Check for the existence of the service principal file, azureauth.json, as set in the azacsnap.json configuration file.

Check the log file, for example, logs/azacsnap-test-azacsnap.log, to see if the service principal file has the correct content. The following log file output shows that the client secret key is invalid.

[19/Nov/2020:18:39:49 +13:00] DEBUG: [PID:0020080:StorageANF:659] [1] Innerexception: Microsoft.IdentityModel.Clients.ActiveDirectory.AdalServiceException AADSTS7000215: Invalid client secret is provided.

Check the log file to see if the service principal has expired. The following log file example shows that the client secret keys are expired.

[19/Nov/2020:18:41:10 +13:00] DEBUG: [PID:0020257:StorageANF:659] [1] Innerexception: Microsoft.IdentityModel.Clients.ActiveDirectory.AdalServiceException AADSTS7000222: The provided client secret keys are expired. Visit the Azure Portal to create new keys for your app, or consider using certificate credentials for added security: https://learn.microsoft.com/azure/active-directory/develop/active-directory-certificate-credentials

Tip

For more information on generating a new Service Principal, refer to the section Enable communication with Storage in the Install Azure Application Consistent Snapshot tool guide.

Troubleshoot failed 'test hana' command

The command azacsnap -c test --test hana might not complete successfully.

Command not found

When setting up communication with SAP HANA, the hdbuserstore program is used to create the secure communication settings. AzAcSnap also requires the hdbsql program for all communications with SAP HANA. These programs are usually under /usr/sap/<SID>/SYS/exe/hdb/ or /usr/sap/hdbclient and must be in the user's $PATH.

In the following example, the hdbsql command isn't in the user's $PATH.

hdbsql -n 172.18.18.50 - i 00 -U AZACSNAP "select version from sys.m_database"

If 'hdbsql' is not a typo you can use command-not-found to lookup the package that contains it, like this:
cnf hdbsql

The following example temporarily adds the hdbsql command to the user's $PATH, allowing azacsnap to run correctly.
```
export PATH=$PATH:/hana/shared/H80/exe/linuxx86_64/hdb/
```

Make sure the installer added the location of these files to the AzAcSnap user's $PATH.

Note

To permanently add to the user's $PATH, update the user's $HOME/.profile file.

Invalid value for key

This command output shows that the connection key hasn't been set up correctly with the hdbuserstore Set command.

hdbsql -n 172.18.18.50 -i 00 -U AZACSNAP "select version from sys.m_database"

* -10104: Invalid value for KEY (AZACSNAP)

For more information on setup of the hdbuserstore, see Get started with AzAcSnap.

Failed test

When validating communication with SAP HANA by running a test with azacsnap -c test --test hana, you might get the following error:

> azacsnap -c test --test hana
BEGIN : Test process started for 'hana'
BEGIN : SAP HANA tests
CRITICAL: Command 'test' failed with error:
Cannot get SAP HANA version, exiting with error: 127

To troubleshoot this error:

Check the configuration file, for example azacsnap.json, for each HANA instance, to ensure that the SAP HANA database values are correct.

Run the following command to verify that the hdbsql command is in the path and that it can connect to the SAP HANA server.

hdbsql -n 172.18.18.50 - i 00 -d SYSTEMDB -U AZACSNAP "\s"

The following example shows the output when the command runs correctly:

host          : 172.18.18.50
sid           : H80
dbname        : SYSTEMDB
user          : AZACSNAP
kernel version: 2.00.040.00.1553674765
SQLDBC version:        libSQLDBCHDB 2.04.126.1551801496
autocommit    : ON
locale        : en_US.UTF-8
input encoding: UTF8
sql port      : saphana1:30013

Insufficient privilege error

If running azacsnap presents an error such as * 258: insufficient privilege, check that the user has the appropriate AZACSNAP database user privileges set up per the installation guide. Verify the user's privileges with the following command:

hdbsql -U AZACSNAP "select GRANTEE,GRANTEE_TYPE,PRIVILEGE,IS_VALID,IS_GRANTABLE from sys.granted_privileges " | grep -i -e GRANTEE -e azacsnap

The command should return the following output:

GRANTEE,GRANTEE_TYPE,PRIVILEGE,IS_VALID,IS_GRANTABLE
"AZACSNAP","USER","BACKUP ADMIN","TRUE","FALSE"
"AZACSNAP","USER","CATALOG READ","TRUE","FALSE"
"AZACSNAP","USER","CREATE ANY","TRUE","TRUE"

The error might provide further information to help determine the required SAP HANA privileges, such as Detailed info for this error can be found with guid '99X9999X99X9999X99X99XX999XXX999' SQLSTATE: HY000. In this case, follow the instructions at SAP Help Portal - GET_INSUFFICIENT_PRIVILEGE_ERROR_DETAILS, which recommend using the following SQL query to determine the details of the required privilege:

CALL SYS.GET_INSUFFICIENT_PRIVILEGE_ERROR_DETAILS ('99X9999X99X9999X99X99XX999XXX999', ?)

GUID,CREATE_TIME,CONNECTION_ID,SESSION_USER_NAME,CHECKED_USER_NAME,PRIVILEGE,IS_MISSING_ANALYTIC_PRIVILEGE,IS_MISSING_GRANT_OPTION,DATABASE_NAME,SCHEMA_NAME,OBJECT_NAME,OBJECT_TYPE
"99X9999X99X9999X99X99XX999XXX999","2021-01-01 01:00:00.180000000",120212,"AZACSNAP","AZACSNAP","DATABASE ADMIN or DATABASE BACKUP ADMIN","FALSE","FALSE","","","",""

In the preceding example, adding the DATABASE BACKUP ADMIN privilege to the SYSTEMDB's AZACSNAP user should resolve the insufficient privilege error.