Troubleshoot Azure Files issues in Linux (SMB)
This article lists common issues that can occur when using SMB Azure file shares with Linux clients. It also provides possible causes and resolutions for these problems.
You can use AzFileDiagnostics to automate symptom detection and ensure that the Linux client has the correct prerequisites. It helps set up your environment to get optimal performance. You can also find this information in the Azure file shares troubleshooter.
Important
This article only applies to SMB shares. For details on NFS shares, see Troubleshoot NFS Azure file shares.
Applies to
File share type | SMB | NFS |
---|---|---|
Standard file shares (GPv2), LRS/ZRS | ||
Standard file shares (GPv2), GRS/GZRS | ||
Premium file shares (FileStorage), LRS/ZRS |
Time stamps were lost when copying files
On Linux/Unix platforms, the cp -p
command fails if different users own file 1 and file 2.
Cause
The force flag f
in COPYFILE results in executing cp -p -f
on Unix. This command also fails to preserve the time stamp of the file that you don't own.
Workaround
Use the storage account user to copy the files:
str_acc_name=[storage account name]
sudo useradd $str_acc_name
sudo passwd $str_acc_name
su $str_acc_name
cp -p filename.txt /share
ls: cannot access '<path>': Input/output error
When you try to list files in an Azure file share by using the ls
command, the command hangs when listing files. You get the following error:
ls: cannot access'<path>': Input/output error
Solution
Upgrade the Linux kernel to the following versions that have a fix for this problem:
- 4.4.87+
- 4.9.48+
- 4.12.11+
- All versions that are greater than or equal to 4.13
Can't create symbolic links - ln: failed to create symbolic link 't': Operation not supported
Cause
By default, mounting Azure file shares on Linux by using SMB doesn't enable support for symbolic links (symlinks). You might see an error like this:
sudo ln -s linked -n t
ln: failed to create symbolic link 't': Operation not supported
Solution
The Linux SMB client doesn't support creating Windows-style symbolic links over the SMB 2 or 3 protocol. Currently, the Linux client supports another style of symbolic links called Minshall+French symlinks for both create and follow operations. Customers who need symbolic links can use the "mfsymlinks" mount option. We recommend "mfsymlinks" because it's also the format that Macs use.
To use symlinks, add the following to the end of your SMB mount command:
,mfsymlinks
So the command looks like the following:
sudo mount -t cifs //<storage-account-name>.file.core.windows.net/<share-name> <mount-point> -o vers=<smb-version>,username=<storage-account-name>,password=<storage-account-key>,dir_mode=0777,file_mode=0777,serverino,mfsymlinks
You can then create symlinks as suggested on the wiki.
Unable to access folders or files which name has a space or a dot at the end
You can't access folders or files from the Azure file share while mounted on Linux. Commands like du and ls and/or third-party applications might fail with a "No such file or directory" error while accessing the share; however, you're able to upload files to these folders via the Azure portal.
Cause
The folders or files were uploaded from a system that encodes the characters at the end of the name to a different character. Files uploaded from a Macintosh computer may have a "0xF028" or "0xF029" character instead of 0x20 (space) or 0X2E (dot).
Solution
Use the mapchars option on the share when mounting the share on Linux:
Instead of:
sudo mount -t cifs $smbPath $mntPath -o vers=3.0,username=$storageAccountName,password=$storageAccountKey,serverino
Use:
sudo mount -t cifs $smbPath $mntPath -o vers=3.0,username=$storageAccountName,password=$storageAccountKey,serverino,mapchars
DNS issues with live migration of Azure storage accounts
File I/Os on the mounted filesystem start giving "Host is down" or "Permission denied" errors. Linux dmesg logs on the client show repeated errors like:
Status code returned 0xc000006d STATUS_LOGON_FAILURE
cifs_setup_session: 2 callbacks suppressed
CIFS VFS: \\contoso.file.core.windows.net Send error in SessSetup = -13
You'll also see that the server FQDN now resolves to a different IP address than the one it's currently connected to. This issue might occur in any scenario where the server IP address can change, such as account migration. Another known scenario is storage account failover because the DNS mapping can change.
Cause
For capacity load balancing purposes, storage accounts are sometimes live-migrated from one storage cluster to another. Account migration triggers Azure Files traffic to be redirected from the source cluster to the destination cluster by updating the DNS mappings to point to the destination cluster. This blocks all traffic to the source cluster from that account. It's expected that the SMB client picks up the DNS updates and redirects further traffic to the destination cluster. However, due to a bug in the Linux SMB kernel client, this redirection doesn't take effect. As a result, the data traffic keeps going to the source cluster, which has stopped serving this account post migration.
Workaround
You can mitigate this issue by rebooting the client OS, but you might run into the issue again if you don't upgrade your client OS to a Linux distro version with account migration support.
While unmounting and remounting the share might seem to resolve the issue temporarily, it's not a permanent solution. When the client reconnects to the server, the issue might occur again. The temporary mitigation occurs because a new mount action bypasses the SMB kernel cache and resolves the DNS address in the user space. However, the kernel DNS cache is utilized during any network disconnection recovery, which can cause the issue to reoccur. This behavior persists even outside of storage account migrations.
To better work around this issue, clear the kernel DNS resolver cache:
Display the status of the kernel
dns_resolver
module by running the following command:grep '.dns_resolver' /proc/keys
You should see command output like the following example:
132b6bbf I------ 1 perm 1f030000 0 0 keyring .dns_resolver: 1
Clear the kernel DNS resolver cache by running the following command:
sudo keyctl clear $((16#$(grep '.dns_resolver' /proc/keys | cut -f1 -d\ ) ))
Display the status of the kernel
dns_resolver
module again:grep '.dns_resolver' /proc/keys
You should see command output like the following example, indicating that the cache is now empty:
132b6bbf I------ 1 perm 1f030000 0 0 keyring .dns_resolver: empty
Unmount and remount the share to mitigate the issue.
Note
On some older Linux distros, the mitigation steps might not work. In such cases, rebooting the client OS will solve the issue temporarily. For a permanent fix, you can add a private endpoint to your storage account and connect to the file share using a private link.
Solution
For a permanent fix, upgrade your client OS to a Linux distro version with account migration support. Several fixes for the Linux SMB kernel client have been submitted to the mainline Linux kernel. The following distros have the fixes:
- Ubuntu: 20.04, 22.04, 24.04, and AKS 22.04 (the fixes are rolled out in kernel version 5.15.0-1068)
- RHEL: 8.6+
- SLES: 15SP2, 15SP3, 15SP4, and 15SP5
- Azure Linux: 2.0 (the fixes are rolled out in kernel version 5.15.159.1) and 3.0
Some distros have backported these fixes. You can check if the following fixes exist in the distro version you're using:
cifs: use the expiry output of dns_query to schedule next resolution
cifs: To match file servers, make sure the server hostname matches
cifs: fix memory leak of smb3_fs_context_dup::server_hostname
dns: Apply a default TTL to records obtained from getaddrinfo()
Unable to mount SMB file share when FIPS is enabled
When Federal Information Processing Standard (FIPS) is enabled in a Linux VM, the SMB file share can't be mounted. The Linux dmesg logs on the client display errors such as:
kernel: CIFS: VFS: Could not allocate crypto hmac(md5)
kernel: CIFS: VFS: Error -2 during NTLMSSP authentication
kernel: CIFS: VFS: \\contoso.file.core.windows.net Send error in SessSetup = -2
kernel: CIFS: VFS: cifs_mount failed w/return code = -2
Important
FIPS is a set of standards that the U.S. government uses to ensure the security and integrity of computer systems. When a system is in FIPS mode, it adheres to specific cryptographic requirements outlined by these standards.
Cause
The client of SMB file share uses the NTLMSSP authentication, which requires the MD5 hashing algorithm. However, in FIPS mode, the MD5 algorithm is restricted because it's not FIPS-compliant. MD5 is a widely used hash function that produces a 128-bit hash value. However, MD5 is considered insecure for cryptographic purposes.
How to check if FIPS mode is enabled
To verify if FIPS mode is enabled on the client, run the following command. If the value is set to 1, then FIPS is enabled.
sudo cat /proc/sys/crypto/fips_enabled
Solution
To resolve this issue, enable Kerberos authentication for SMB file share. If FIPS is enabled unintentionally, refer to option2 to disable it.
Option 1: Enable Kerberos authentication for SMB file share
To mount a SMB file share on the Linux VM where FIPS is enabled, use Kerberos/Azure AD authentication. For more information, see Enable Active Directory authentication over SMB for Linux clients accessing Azure Files.
Option 2: Disable FIPS to mount the Samba share
Change the sysctl value of
crypto.fips_enabled
to 0 in/etc/sysctl.conf
.Modify the
GRUB_CMDLINE_LINUX_DEFAULT
in/etc/default/grub
file and remove the parameterfips=1
.Rebuilt the grub2 config file with the following command:
sudo grub2-mkconfig -o /boot/grub2/grub.cfg
Rebuild the initramfs image with the following command:
sudo dracut -fv
Reboot the VM.
For more information, see to the following documents from Linux distributors:
- RedHat: Why would enabling FIPS mode in the kernel break CIFS mounts
- SUSE: CIFS mount fails with error "mount error(2): No such file or directory"
Need help?
If you still need help, contact support to get your problem resolved quickly.
See also
- Troubleshoot Azure Files
- Troubleshoot Azure Files performance
- Troubleshoot Azure Files connectivity (SMB)
- Troubleshoot Azure Files authentication and authorization (SMB)
- Troubleshoot Azure Files general NFS issues on Linux
- Troubleshoot Azure File Sync issues
Contact us for help
If you have questions or need help, create a support request, or ask Azure community support. You can also submit product feedback to Azure feedback community.