Memory error when using blobfuse2 to make a new mount

Question

A month ago I mounted my first blob storage on my Linux VM using blobfuse2. The SAS token went out of date and I believed that I had to redo the mount to update the token. I removed the blob folder on my VM (I should probably have used blobfuse2 unmount instead). When I tried to redo the mount I got the following error:

Error: failed to initialize new pipeline [not enough free memory for provided stream configuration]

The full log is here:

> Feb 19 14:53:48 kahy-cpc blobfuse2[3737]: LOG_CRIT [mount.go (404)]: Starting Blobfuse2 Mount : 2.2.0 on [Ubuntu 22.04.3 LTS]
> Feb 19 14:53:48 kahy-cpc blobfuse2[3737]: LOG_CRIT [mount.go (405)]: Logging level set to : LOG_DEBUG
> Feb 19 14:53:48 kahy-cpc blobfuse2[3737]: LOG_DEBUG [mount.go (406)]: Mount allowed on nonempty path : false
> Feb 19 14:53:48 kahy-cpc blobfuse2[3737]: LOG_TRACE [libfuse.go (253)]: Libfuse::Configure : libfuse
> Feb 19 14:53:48 kahy-cpc blobfuse2[3737]: LOG_INFO [libfuse.go (244)]: Libfuse::Validate : UID 0, GID 0
> Feb 19 14:53:48 kahy-cpc blobfuse2[3737]: LOG_INFO [libfuse.go (305)]: Libfuse::Configure : read-only false, allow-other false, allow-root false, default-perm 493, entry-timeout 120, attr-time 120, negative-tim>Feb 19 14:53:48 kahy-cpc blobfuse2[3737]: LOG_TRACE [stream.go (100)]: Stream::Configure : stream
> Feb 19 14:53:48 kahy-cpc blobfuse2[3737]: LOG_ERR [stream.go (127)]: Stream::Configure : config error, not enough free memory for provided configuration
> Feb 19 14:53:48 kahy-cpc blobfuse2[3737]: LOG_ERR [pipeline.go (69)]: Pipeline: error creating pipeline component stream [not enough free memory for provided stream configuration]
> Feb 19 14:53:48 kahy-cpc blobfuse2[3737]: LOG_ERR [mount.go (409)]: mount : failed to initialize new pipeline [not enough free memory for provided stream configuration]

I used the same buffer specs as before and even tried to reduce them, but without success. My streamConfig.yaml is this:

  GNU nano 6.2                                                                                     streamConfig.yaml                                                                                               
logging:
  type: syslog
  level: log_debug
components:
  - libfuse
  - stream
  - attr_cache
  - azstorage
libfuse:
  attribute-expiration-sec: 120
  entry-expiration-sec: 120
  negative-entry-expiration-sec: 240
stream:
  block-size-mb: 8
  max-buffers: 42
  buffer-size-mb: 4096
attr_cache:
  timeout-sec: 7200
azstorage:
  type: block
  account-name: <account-name>
  sas: <sas token>
  endpoint: <endpoint>
  mode: sas
  container: blob-folder

Any tips on what I can do to successfully re-mount the blob?

Answer 1

Hello Kari,

Greetings! Welcome to Microsoft Q&A Forum.

It seems you encountered an issue while remounting your blob storage using blobfuse2 on your Linux VM. The error message you received, “failed to initialize new pipeline [not enough free memory for provided stream configuration],” indicates that there might be insufficient memory available for the stream configuration.

1.Ensure that your VM have sufficient memory available, you can check with free -h command in terminal as well.

2.Sometimes, the Linux kernel’s OOM Killer terminates processes when the system is critically low on memory.

• To check if Blobfuse2 was killed due to memory constraints, run: dmesg | grep -i oom.
• If Blobfuse2 was killed, consider allocating more memory to your VM or optimizing memory usage.

Also, just wanted to check with you if you have followed the below steps to persist mount with BlobFuse2 lib mentioned in the article: https://github.com/Azure/azure-storage-fuse/wiki/Blobfuse2-Installation#persisting-mount

1. Make sure the fuse package is installed (e.g., yum install fuse3 / apt-get install fuse3)
2. Update config.yaml file with your preferred configuration.
3. Edit /etc/fstab with the blobfuse script.

Add the following line to use mount.sh:

YAML

/<path_to_blobfuse2_mount.sh_file>/mount.sh </path/to/desired/mountpoint> fuse defaults,_netdev 0 0

OR

Add the following line to run without mount.sh

YAML

blobfuse2 /home/azureuser/mntblobfuse fuse defaults,_netdev,--config-file=/home/azureuser/config.yaml,allow_other 0 0

refer - https://learn.microsoft.com/en-us/troubleshoot/azure/azure-kubernetes/mounting-azure-blob-storage-container-fail for more information on troubleshooting the issue and https://learn.microsoft.com/en-us/azure/storage/blobs/storage-how-to-mount-container-linux?tabs=RHEL

Hope this answer helps! Please let us know if you have any further queries. I’m happy to assist you further.  

 

Answer 2

I ran into the same problem. In the config above

stream: block-size-mb: 8

max-buffers: 42

buffer-size-mb: 4096

The mount will try to reserve 42 buffers of 4096mb each. The server would need at least that much available RAM to successfully complete the mount. You can tune the amount of buffers, and the size of each buffer to fit the available RAM.