Issue accessing/saving files into Azure Storage Account using AKS application

Question

I have a multiple similar application deployed on Kubernetes. They are .net core applications and one of the functionalities is to save mysql database backup or files to Azure Storage account.

This has recently became an issue. In some cases but not all the application has access denied to Storage Account. The exception message looks like this :
Access to the path '/app/CustomerFileShare/BackupDatabase/backup_2024031409490647.sql' is denied.System.UnauthorizedAccessException: Access to the path '/app/CustomerFileShare/BackupDatabase/backup_2024031409490647.sql' is denied. ---> System.IO.IOException: Permission denied .

My yaml related config :

spec:
      containers:
        ...
        volumeMounts:
        - name: customerfiles
          mountPath: /app/CustomerFileShare
       ...
        env:
        ...
        - name: FILE_SHARE_PATH
          value: "/app/CustomerFileShare"
        ...
      volumes:
      - name: customerfiles
        azureFile:
          secretName: webstorage-test
          shareName: customerfiles
          readOnly: false

I have looked into service principle role assignment and couldn't find any issue there that could cause it. The applications have "Contributor" role assigned in all cases.

Same goes for Secrets and Connection strings. I tried creating new ones, then lso inspected the content of the secret and it matched with the AccessKey of the Storage Account. This didn't do anything.
The yaml file that I use to configure everything was also unchanged between. The apps that can save files and can't have the same setup that should allow for writing into storage account.

Anything else that I could look into? It does look very random to me that suddenly in the last 3 weeks I can't save files and the code in the application has been unchanged for long time and didnt have issues before.

Answer 1

Hello ERAS2 Hamburg, thank you for posting your query here!

Firstly, apologies for the delay in response.

I understand that you're encountering intermittent access denied errors to Azure Storage when running .NET Core applications in Kubernetes.

The error message indicates a file system permission issue. It’s possible that the application does not have the necessary permissions to write to the specified path in the container. You could try using an initContainer to change the ownership or permissions of the /app/CustomerFileShare directory before the application container starts.

You could set the fsGroup in the pod’s security context. This will change the ownership of the volume to the specified group ID, and any processes in the pod that run with this group ID will be able to write to the volume.

Also, please verify if the keys for the Azure Storage account were rotated recently and if the Kubernetes secret was updated accordingly.

Do let us know if you have any further queries. I’m happy to assist you further.