Blob Container is mounted on 2 server using blobfuse2. The file written on server not being shown on another.

Question

I have created a blob container to store user-uploaded files. I have 2 servers running using AFD. The blob container has been mounted using blobfuse2 (the config file is attached below).

So, my web application processes the file and generates the log file for the file uploaded by the user. However, I have observed that the log file content is different on the two servers, even though they have been using the same blob container.

For example, If the user is connected to server 1, the log file is complete on server 1 and blob storage (verified from the Azure portal), however, server 2 shows the complete log file. I have waited for hours to see if there would be any sync happening later but it never does. It is not for all the processes, but only for some.

Any solution, tips, or workaround would be helpful.

Command to Mount Blob Container

sudo blobfuse2 mount /path/to/dir/ --config-file=config.yaml

logging:
  type: syslog
  level: log_debug
  track-time: true
components:
  - libfuse
  - stream
  - attr_cache
  - azstorage
libfuse:
  attribute-expiration-sec: 120
  entry-expiration-sec: 120
  negative-entry-expiration-sec: 240
stream:
  block-size-mb: 8
  max-buffers: 64
  buffer-size-mb: 36
attr_cache:
  timeout-sec: 7200
azstorage:
  type: adls
  account-name: xxxxxxxx
  account-key: xxxxxxxx
  endpoint: https://xxxxxxxxx
  mode: key
  container: xxxxxxxx
health_monitor:
  enable-monitoring: true
  stats-poll-interval-sec: 10

Answer 1

Hi Gopal Chandra Nepal - Thanks for reaching out over Q&A forum..

From the config, it appears you are mainly using block cache rather than a file cache as the configuration option.

Can you try using block cache method once and set the cache timeout to 0 and observe ahead.

Answer 2

Hi @Gopal Chandra Nepal

Greetings! Welcome to Microsoft Q&A Forum and thanks for posting your query here!

In addition to the above solution from Amrinder Singh. We would suggest you follow the below.

I understand that you are experiencing an issue with file synchronization between two servers that are both mounting the same Blob Storage container using Blobfuse2.

It is important to note that when using blobfuse2 to mount blob container on multiple servers it does not support overlapping mouth paths. If both servers are mounting the same blob container, make sure that you are using unique and non-overlapping mount points.

Blobfuse2 is not designed for multiple server access to the same blob container. If the log files are showing different content on the two servers, it could be due to caching issues or delays in synchronization. BlobFuse2 uses local file caching techniques, and changes made on one server may not immediately reflect on the other server if they are accessing the same blob container simultaneously.

Here are the few possible ways where we can try fixing the issue.

• You can adjust the cache settings in your configuration file to reduce the cache expiration times. For example, you can set attr_cache and entry-expiration-sec to lower values to ensure that metadata is refreshed more frequently. By setting the timeout to 0 in the libfuse section and removing the attr_cache from the config file can help resolve synchronization issues.
• Also, please make sure that health monitoring is working correctly from the above config file which says that it is already enabled, but make sure to check the logs for any issues that might indicate problems with the connectivity or synchronization.
• If possible, implement a mechanism to refresh or clear the cache on the servers to ensure they are accessing the latest data.
• Also, there might still be some latency in how quickly changes are reflected across different mounts. Testing with smaller files or simpler operations might help determine if this is the case.

I hope this helps in solving the issue by changing the cache settings and reviewing the configuration settings for Blobfuse2 and ensure that both the servers are using the same version of Blobfuse2.

Please 'Upvote'(Thumbs-up) and 'Accept' as an answer if the reply was helpful. This will benefit other community members who face the same issue.

Answer 3

Hello Amrider and Hari,

I tried setting attr_cache to 0 and also 1. I also did set entry-expiration-sec to 0, however the issue is still the same.

I have observed the issue happens mainly when a user uploads data on server 1 but initate's process/makes changes from server 2. My servers are connected via AFD and usea round-robin approach.