Troubleshoot the Azure Application Consistent Snapshot (AzAcSnap) tool
This article describes how to troubleshoot issues when using the Azure Application Consistent Snapshot (AzAcSnap) tool for Azure NetApp Files and Azure Large Instance.
You might encounter several common issues when running AzAcSnap commands. Follow the instructions to troubleshoot the issues. If you still have issues, open a Service Request for Microsoft Support from the Azure portal and assign the request to the SAP HANA Large Instance queue.
AzAcSnap command won't run
In some cases AzAcSnap won't start due to the user's environment.
Failed to create CoreCLR
AzAcSnap is written in .NET and the CoreCLR is an execution engine for .NET apps, performing functions such as IL byte code loading, compilation to machine code and garbage collection. In this case there is an environmental problem blocking the CoreCLR engine from starting.
A common cause is limited permissions or environmental setup for the AzAcSnap operating system user, usually 'azacsnap'.
Failed to create CoreCLR, HRESULT: 0x80004005 can be caused by lack of write access for the azacsnap user to the system's
All command lines starting with
# are commands run as
root, all command lines starting with
> are run as
/tmp ownership and permissions (note in this example only the
root user can read and write to
# ls -ld /tmp drwx------ 9 root root 8192 Mar 31 10:50 /tmp
/tmp has the following permissions, which would allow the azacsnap user to run the azacsnap command:
# ls -ld /tmp drwxrwxrwt 9 root root 8192 Mar 31 10:51 /tmp
If it's not possible to change the
/tmp directory permissions, then create a user specific
TMPDIR for the
> mkdir /home/azacsnap/_tmp > export TMPDIR=/home/azacsnap/_tmp > azacsnap -c about
WKO0XXXXXXXXXXXNW Wk,.,oxxxxxxxxxxx0W 0;.'.;dxxxxxxxxxxxKW Xl'''.'cdxxxxxxxxxdkX Wx,''''.,lxxxxdxdddddON 0:''''''.;oxdddddddddxKW Xl''''''''':dddddddddddkX Wx,''''''''':ddddddddddddON O:''''''''',xKxddddddoddod0W Xl''''''''''oNW0dooooooooooxX Wx,,,,,,'','c0WWNkoooooooooookN WO:',,,,,,,,;cxxxxooooooooooooo0W Xl,,,,,,,;;;;;;;;;;:llooooooooldX Nx,,,,,,,,,,:c;;;;;;;;coooollllllkN WO:,,,,,,,,,;kXkl:;;;;,;lolllllllloOW Xl,,,,,,,,,,dN WNOl:;;;;:lllllllllldK 0c,;;;;,,,;lK NOo:;;:clllllllllo0W WK000000000N NK000KKKKKKKKKKXW Azure Application Consistent Snapshot Tool AzAcSnap 7a (Build: 1AA8343)
Changing the user's
TMPDIR would need to be made permanent by changing the user's profile (e.g.
$HOME/.bash_profile). There would also be a need to clean-up the
TMPDIR on system reboot, this is typically automatic for
Check log files, result files, and syslog
Some of the best sources of information for investigating AzAcSnap issues are the log files, result files, and the system log.
The AzAcSnap log files are stored in the directory configured by the
logPath parameter in the AzAcSnap configuration file. The default configuration filename is azacsnap.json, and the default value for
logPath is ./logs, which means the log files are written into the ./logs directory relative to where the
azacsnap command runs. If you make the
logPath an absolute location, such as /home/azacsnap/logs,
azacsnap always outputs the logs into /home/azacsnap/logs, regardless of where you run the
The log filename is based on the application name,
azacsnap, the command run with
-c, such as
details, and the default configuration filename, such as azacsnap.json. With the
-c backup command, a default log filename would be azacsnap-backup-azacsnap.log, written into the directory configured by
This naming convention allows for multiple configuration files, one per database, to help locate the associated log files. If the configuration filename is SID.json, then the log filename when using the
azacsnap -c backup --configfile SID.json option is azacsnap-backup-SID.log.
Result files and syslog
-c backup command, AzAcSnap writes to a *.result file. The purpose of the *.result file is to provide high-level confirmation of success/failure. If the *.result file is empty, then assume failure. Any output written to the *.result file is also output to the system log (for example,
/var/log/messages) by using the
logger command. The *.result filename has the same base name as the log file to allow for matching the result file with the configuration file and the backup log file. The *.result file goes into the same location as the other log files and is a simple one line output file.
Example for successful completion:
Output to *.result file:
Database # 1 (PR1) : completed ok
Dec 17 09:01:13 azacsnap-rhel azacsnap: Database # 1 (PR1) : completed ok
Example output where a failure has occured and AzAcSnap captured the failure:
Output to *.result file:
Database # 1 (PR1) : failed
Dec 19 09:00:30 azacsnap-rhel azacsnap: Database # 1 (PR1) : failed
Troubleshoot failed 'test storage' command
azacsnap -c test --test storage might not complete successfully.
Check network firewalls
Communication with Azure NetApp Files might fail or time out. To troubleshoot, make sure firewall rules aren't blocking outbound traffic from the system running AzAcSnap to the following addresses and TCP/IP ports:
Use Cloud Shell to validate configuration files
You can test whether the service principal is configured correctly by using Cloud Shell through the Azure portal. Using Cloud Shell tests for correct configuration, bypassing network controls within a virtual network or virtual machine (VM).
In the Azure portal, open a Cloud Shell session.
Make a test directory, for example
Switch to the azacsnap directory, and download the latest version of AzAcSnap.
Make the installer executable, for example
chmod +x azacsnapinstaller.
Extract the binary for testing.
./azacsnapinstaller -X -d .
The results look like the following output:
+-----------------------------------------------------------+ | Azure Application Consistent Snapshot Tool Installer | +-----------------------------------------------------------+ |-> Installer version '5.0.2_Build_20210827.19086' |-> Extracting commands into .. |-> Cleaning up .NET extract dir
Use the Cloud Shell Upload/Download icon to upload the service principal file, azureauth.json, and the AzAcSnap configuration file, such as azacsnap.json, for testing.
./azacsnap -c test --test storage
The test command can take about 90 seconds to complete.
Failed test on Azure Large Instance
The following error example is from running
azacsnap on Azure Large Instance:
azacsnap -c test --test storage
The authenticity of host '172.18.18.11 (172.18.18.11)' can't be established. ECDSA key fingerprint is SHA256:QxamHRn3ZKbJAKnEimQpVVCknDSO9uB4c9Qd8komDec. Are you sure you want to continue connecting (yes/no)?
To troubleshoot this error, don't respond
yes. Make sure that your storage IP address is correct. You can confirm the storage IP address with the Microsoft operations team.
The error usually appears when the Azure Large Instance storage user doesn't have access to the underlying storage. To determine whether the storage user has access to storage, run the
ssh command to validate communication with the storage platform.
ssh <StorageBackupname>@<Storage IP address> "volume show -fields volume"
The following example shows the expected output:
ssh firstname.lastname@example.org "volume show -fields volume"
vserver volume --------------------------------- ------------------------------ osa33-hana-c01v250-client25-nprod hana_data_h80_mnt00001_t020_vol osa33-hana-c01v250-client25-nprod hana_data_h80_mnt00002_t020_vol
Failed test with Azure NetApp Files
The following error example is from running
azacsnap with Azure NetApp Files:
azacsnap --configfile azacsnap.json.NOT-WORKING -c test --test storage
BEGIN : Test process started for 'storage' BEGIN : Storage test snapshots on 'data' volumes BEGIN : 1 task(s) to Test Snapshots for Storage Volume Type 'data' ERROR: Could not create StorageANF object [authFile = 'azureauth.json']
To troubleshoot this error:
Check for the existence of the service principal file, azureauth.json, as set in the azacsnap.json configuration file.
Check the log file, for example, logs/azacsnap-test-azacsnap.log, to see if the service principal file has the correct content. The following log file output shows that the client secret key is invalid.
[19/Nov/2020:18:39:49 +13:00] DEBUG: [PID:0020080:StorageANF:659]  Innerexception: Microsoft.IdentityModel.Clients.ActiveDirectory.AdalServiceException AADSTS7000215: Invalid client secret is provided.
Check the log file to see if the service principal has expired. The following log file example shows that the client secret keys are expired.
[19/Nov/2020:18:41:10 +13:00] DEBUG: [PID:0020257:StorageANF:659]  Innerexception: Microsoft.IdentityModel.Clients.ActiveDirectory.AdalServiceException AADSTS7000222: The provided client secret keys are expired. Visit the Azure Portal to create new keys for your app, or consider using certificate credentials for added security: https://learn.microsoft.com/azure/active-directory/develop/active-directory-certificate-credentials
For more information on generating a new Service Principal, refer to the section Enable communication with Storage in the Install Azure Application Consistent Snapshot tool guide.
Troubleshoot failed 'test hana' command
azacsnap -c test --test hana might not complete successfully.
Command not found
When setting up communication with SAP HANA, the
hdbuserstore program is used to create the secure communication settings. AzAcSnap also requires the
hdbsql program for all communications with SAP HANA. These programs are usually under /usr/sap/<SID>/SYS/exe/hdb/ or /usr/sap/hdbclient and must be in the user's
In the following example, the
hdbsqlcommand isn't in the user's
hdbsql -n 172.18.18.50 - i 00 -U AZACSNAP "select version from sys.m_database"
If 'hdbsql' is not a typo you can use command-not-found to lookup the package that contains it, like this: cnf hdbsql
The following example temporarily adds the
hdbsqlcommand to the user's
azacsnapto run correctly.
Make sure the installer added the location of these files to the AzAcSnap user's
To permanently add to the user's
$PATH, update the user's $HOME/.profile file.
Invalid value for key
This command output shows that the connection key hasn't been set up correctly with the
hdbuserstore Set command.
hdbsql -n 172.18.18.50 -i 00 -U AZACSNAP "select version from sys.m_database"
* -10104: Invalid value for KEY (AZACSNAP)
For more information on setup of the
hdbuserstore, see Get started with AzAcSnap.
When validating communication with SAP HANA by running a test with
azacsnap -c test --test hana, you might get the following error:
> azacsnap -c test --test hana BEGIN : Test process started for 'hana' BEGIN : SAP HANA tests CRITICAL: Command 'test' failed with error: Cannot get SAP HANA version, exiting with error: 127
To troubleshoot this error:
Check the configuration file, for example azacsnap.json, for each HANA instance, to ensure that the SAP HANA database values are correct.
Run the following command to verify that the
hdbsqlcommand is in the path and that it can connect to the SAP HANA server.
hdbsql -n 172.18.18.50 - i 00 -d SYSTEMDB -U AZACSNAP "\s"
The following example shows the output when the command runs correctly:
host : 172.18.18.50 sid : H80 dbname : SYSTEMDB user : AZACSNAP kernel version: 2.00.040.00.1553674765 SQLDBC version: libSQLDBCHDB 2.04.126.1551801496 autocommit : ON locale : en_US.UTF-8 input encoding: UTF8 sql port : saphana1:30013
Insufficient privilege error
azacsnap presents an error such as
* 258: insufficient privilege, check that the user has the appropriate AZACSNAP database user privileges set up per the installation guide. Verify the user's privileges with the following command:
hdbsql -U AZACSNAP "select GRANTEE,GRANTEE_TYPE,PRIVILEGE,IS_VALID,IS_GRANTABLE from sys.granted_privileges " | grep -i -e GRANTEE -e azacsnap
The command should return the following output:
GRANTEE,GRANTEE_TYPE,PRIVILEGE,IS_VALID,IS_GRANTABLE "AZACSNAP","USER","BACKUP ADMIN","TRUE","FALSE" "AZACSNAP","USER","CATALOG READ","TRUE","FALSE" "AZACSNAP","USER","CREATE ANY","TRUE","TRUE"
The error might provide further information to help determine the required SAP HANA privileges, such as
Detailed info for this error can be found with guid '99X9999X99X9999X99X99XX999XXX999' SQLSTATE: HY000. In this case, follow the instructions at SAP Help Portal - GET_INSUFFICIENT_PRIVILEGE_ERROR_DETAILS, which recommend using the following SQL query to determine the details of the required privilege:
CALL SYS.GET_INSUFFICIENT_PRIVILEGE_ERROR_DETAILS ('99X9999X99X9999X99X99XX999XXX999', ?)
GUID,CREATE_TIME,CONNECTION_ID,SESSION_USER_NAME,CHECKED_USER_NAME,PRIVILEGE,IS_MISSING_ANALYTIC_PRIVILEGE,IS_MISSING_GRANT_OPTION,DATABASE_NAME,SCHEMA_NAME,OBJECT_NAME,OBJECT_TYPE "99X9999X99X9999X99X99XX999XXX999","2021-01-01 01:00:00.180000000",120212,"AZACSNAP","AZACSNAP","DATABASE ADMIN or DATABASE BACKUP ADMIN","FALSE","FALSE","","","",""
In the preceding example, adding the
DATABASE BACKUP ADMIN privilege to the SYSTEMDB's AZACSNAP user should resolve the insufficient privilege error.