how to refresh the files from the storage account mounted using the path mapping

Aruna Ganguri 0 Reputation points
2023-10-03T13:14:23.7233333+00:00

I have mounted the Azure data lake storage containers in python azure webapp as path mapping. When reading the files, the files are not up to date. The files on the mounted path are visible immediately after the initial creation, but next changes are refreshing after around 4 minutes(240 seconds). How to get the files updated on the mounted path immediately

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,378 questions
Azure App Service
Azure App Service
Azure App Service is a service used to create and deploy scalable, mission-critical web apps.
7,052 questions
Azure Static Web Apps
Azure Static Web Apps
An Azure service that provides streamlined full-stack web app development.
787 questions
{count} votes

1 answer

Sort by: Most helpful
  1. BhargavaGunnam-MSFT 27,896 Reputation points Microsoft Employee
    2023-10-03T20:10:57.5466667+00:00

    Hello Aruna Ganguri,

    Welcome to the Microsoft Q&A forum.

    It looks like you are experiencing a delay in refreshing the mounted Azure Data Lake Storage Gen1 files on your Python Azure webapp.

    In general, When you mount Azure Data Lake Storage containers as path mapping in a Python Azure Web App, the files may not be updated immediately due to caching.

    By default, the Azure Web App uses a caching mechanism to improve performance and reduce the number of requests to the mounted path.

    To get the files updated on the mounted path immediately, you can disable caching by setting the WEBSITE_LOCAL_CACHE_OPTION

    Please see the below document for more details.

    https://learn.microsoft.com/en-us/azure/app-service/reference-app-settings?tabs=kudu%2Cdotnet

    also, please check if you can use dbutils.fs.refreshMounts() command to refresh all the mount points on your webapp.

    This command will force the webapp to sync with the latest state of the storage account and update the files

    https://learn.microsoft.com/en-us/azure/databricks/dbfs/mounts

    I hope this helps. Please let me know if you have any further questions.

    0 comments No comments