how to refresh the files from the storage account mounted using the path mapping

Aruna Ganguri 0

I have mounted the Azure data lake storage containers in python azure webapp as path mapping. When reading the files, the files are not up to date. The files on the mounted path are visible immediately after the initial creation, but next changes are refreshing after around 4 minutes(240 seconds). How to get the files updated on the mounted path immediately

Aruna Ganguri 0 Reputation points

2023-10-04T14:44:37.88+00:00

i have set the application setting for WEBSITE_LOCAL_CACHE_OPTION as Disabled. But still the file is not getting refreshed immediately. when i open the SSH for the webapp and keep monitoring the file, i observe the file permission change from root/root to nobody/nogroup and finally back to root/root and getting updated at around 230 seconds
Bhargava-MSFT 29,266 Reputation points Microsoft Employee

2023-10-04T22:36:39.89+00:00

Hello Aruna Ganguri,

If you have already disabled the local cache option and are still experiencing a delay in refreshing the mounted files, it is possible that the issue is related to the file permissions on the mounted path.

As the file permission changes from root/root to nobody/nogroup and back to root/root, so it seems the web app uses a temporary storage account to cache the files from the mounted path. This storage account may have different permissions than the Data Lake Storage Gen1 account and may cause a conflict when syncing the files.

Can you change the file permissions on the mounted path to nobody/nogroup and see if it helps
Aruna Ganguri 0 Reputation points

2023-10-05T17:06:52.1466667+00:00

I see the mounted folders are with nobody/nogroup and the files within the folder are in root/root permissions. Are you asking me to change the folder permissions or each file permission?

As you mentioned the permission changes clearly states that there is some temporary storage which is caching. Even then, the refresh is still happening after a delay. So i am not sure changing the file permissions would refresh it faster.
Bhargava-MSFT 29,266 Reputation points Microsoft Employee

2023-10-05T22:40:47.3133333+00:00

Thanks for the additional details. It seems like the issue may be related to the temporary storage that is caching the files.

Can you check if you can force the web app to retrieve the latest version of the files by clearing the cache for the mounted path?
Bhargava-MSFT 29,266 Reputation points Microsoft Employee

2023-10-10T17:50:06.8766667+00:00

Hello Aruna Ganguri,

I am checking to see if the above suggestion is helpful.
Aruna Ganguri 0 Reputation points

2023-10-12T03:28:34.0133333+00:00

i am not sure how to force the webapp to get the latest version. Whenever a new file is created on the mounted path, the file is available on the mounted path immediately. Only the subsequent writes on the file are not getting refreshed immediately. As a work around i used the DataLakeFileClient to connect to the storage account and read the file directly. I would prefer if the file on the mounted path gets refreshed immediately.
Bhargava-MSFT 29,266 Reputation points Microsoft Employee

2023-10-12T17:16:39.4166667+00:00

did you get a chance to use dbutils.fs.refreshMounts() command to refresh all the mount points on your webapp code(as mentioned in my previous comment)

This command will force the web app to sync with the latest state of the storage account and update the files.

1 answer

Bhargava-MSFT 29,266 Reputation points Microsoft Employee

2023-10-03T20:10:57.5466667+00:00

Hello Aruna Ganguri,

Welcome to the Microsoft Q&A forum.

It looks like you are experiencing a delay in refreshing the mounted Azure Data Lake Storage Gen1 files on your Python Azure webapp.

In general, When you mount Azure Data Lake Storage containers as path mapping in a Python Azure Web App, the files may not be updated immediately due to caching.

By default, the Azure Web App uses a caching mechanism to improve performance and reduce the number of requests to the mounted path.

To get the files updated on the mounted path immediately, you can disable caching by setting the WEBSITE_LOCAL_CACHE_OPTION

Please see the below document for more details.

https://learn.microsoft.com/en-us/azure/app-service/reference-app-settings?tabs=kudu%2Cdotnet

also, please check if you can use dbutils.fs.refreshMounts() command to refresh all the mount points on your webapp.

This command will force the webapp to sync with the latest state of the storage account and update the files

https://learn.microsoft.com/en-us/azure/databricks/dbfs/mounts

I hope this helps. Please let me know if you have any further questions.
Please sign in to rate this answer.

0 comments No comments
Sign in to comment

Share via

how to refresh the files from the storage account mounted using the path mapping

1 answer