Managing Azure FIle Share through app service SSH -- Corruption

Question

I have an Azure file share mounted on a linux app service. I am trying to compress some large directories (~ 10,000 files / 17g) by running tar -z through an ssh connection to the app service. This has worked several times.

Now I am getting some signs that the file-system is corrupt. For example, tar has replied with

root@myapp:/var/www/html/scans# tar czf CDB240611.tgz CDB240611
gzip: stdout: Input/output error
tar: Child returned status 1
tar: Error is not recoverable: exiting now

Two questions:

1. Is there any way to repair the filesystem?
2. Is there a way to do this sort of compression within the azure cloud without effectively downloading all the files, compressing them and re-uploading them?

Thank you.

Answer 1

Hello Paul Cote,

Welcome to the Microsoft Q&A and thank you for posting your questions here.

Problem

I understand that you would like to repair filesystem and perform compression within the azure cloud without effectively downloading all the files.

Solution

Azure file shares are managed services, so traditional filesystem repair tools like fsck are not applicable directly. However, here are a few steps to troubleshoot and potentially resolve the issue:

1. Verify if there are any ongoing issues with Azure File Storage in the region where your file share is hosted. You can check the Azure status page for any reported problems.
2. Sometimes, simply unmounting and remounting the file share can help resolve transient issues. You can do this by unmounting the file share in your Linux app service and then remounting it. https://docs.microsoft.com/en-us/azure/storage/files/storage-files-introduction
3. If the issue persists, consider running a file integrity check on your files. You can use tools like md5sum to ensure that files are not corrupted.

Secondly, about the second question, to avoid downloading and uploading files for compression, you can use Azure-native solutions to handle the compression within the cloud. These are solution that you can use for your need:

1. You can create a batch job that compresses the files directly within Azure. Here’s a high-level overview of how you can do this:
   1. Create an Azure Batch account.
   2. Create a pool of compute nodes.
   3. Create a job with tasks that execute the tar command to compress files on the nodes.
   4. https://docs.microsoft.com/en-us/azure/batch/
2. For smaller scale or more event-driven tasks, Azure Functions can be used to run compression tasks. You can trigger an Azure Function that runs a script to compress the files.
   • Create an Azure Function with a timer or HTTP trigger.
   • Use the Azure Storage SDK within the function to access the file share.
   • Run the compression logic using a language supported by Azure Functions (e.g., Python, PowerShell).
   • Example of the code in python will look similar to the below:

          import os
          import tarfile
          from azure.storage.fileshare import ShareFileClient
          def compress_files(directory_path, output_filename):
              with tarfile.open(output_filename, "w:gz") as tar:
                  tar.add(directory_path, arcname=os.path.basename(directory_path))
          def main():
              directory_path = "/path/to/your/files"
              output_filename = "output.tgz"
              
              compress_files(directory_path, output_filename)
              # Upload the compressed file back to Azure File Share
              file_client = ShareFileClient.from_connection_string(conn_str="your_connection_string", share_name="your_share_name", file_path=output_filename)
              with open(output_filename, "rb") as source_file:
                  file_client.upload_file(source_file)
          if __name__ == "__main__":
              main()
     

     https://docs.microsoft.com/en-us/azure/azure-functions/ and https://docs.microsoft.com/en-us/python/api/overview/azure/storage?view=azure-python
3. For a more automated and codeless approach, Azure Logic Apps can be used to orchestrate the compression process.

• Create a Logic App that triggers based on a schedule or event.
• Use Azure Storage connectors to access files.
  • Execute compression using built-in actions or Azure Functions. https://docs.microsoft.com/en-us/azure/logic-apps/

1. If you need more control, you can use an Azure VM or container to handle the compression. Mount the Azure file share on the VM or container, and run your compression scripts as you would on a local machine. https://docs.microsoft.com/en-us/azure/virtual-machines

References

Kindly use the links provided as additional resources for your need.

Accept Answer

I hope this is helpful! Do not hesitate to let me know if you have any other questions.

** Please don't forget to close up the thread here by upvoting and accept it as an answer if it is helpful ** so that others in the community facing similar issues can easily find the solution.

Best Regards,

Sina Salam