Edit

Use Azure Blob storage with Azure Managed Lustre

  • Article

This article explains concepts for using the Azure blob integration with Azure Managed Lustre file systems.

To learn the requirements and configuration needed for a compatible blob container, Blob integration prerequisites.

Azure Managed Lustre supports both hierarchical and non-hierarchical namespaces for blobs with the following minor differences:

  • With a hierarchical namespace container, Azure Managed Lustre reads POSIX attributes from the blob header.
  • With a non-hierarchical container, Azure Managed Lustre reads POSIX attributes from the blob metadata. A separate empty file with the same name as your blob container contents is created to hold the metadata. This file is a sibling to the actual data directory in the Azure Managed Lustre file system.

Some key interactions between Azure Managed Lustre and Blob include:

  • Only file names (namespace) and metadata are imported when the file system is created. The Blob contents are imported later when first accessed.

    At file system creation time, the namespace and metadata from the blob or HNS storage container are imported into the Lustre namespace.  Each blob is imported as one file with slashes in the blob path names corresponding to directories created in Lustre.  You may specify a subset of the namespace to import using an import prefix.

  • Upon first client access (after file system deployment is completed), the contents of the blobs are pulled into the file system.  There will be a slight delay upon first access until the data is available while the Lustre Hierarchical Storage Management (HSM) feature pulls in the blob contents to the corresponding file in the file system.

  • As new files are created or existing files are modified in the file system, you can export these files back to the storage account by running Lustre cli commands on the client, or using the managed process as described in the Export data using archive jobs article.

Note

You can also prefetch the contents of blobs using Lustre's lfs hsm_restore command from a mounted client with sudo capabilities using the command below.

nohup find local/directory -type f -print0 | xargs -0 -n 1 sudo lfs hsm_restore &

Filter blob imports

When you create the Azure Managed Lustre file system, you can specify a prefix to filter the data imported into the Azure Managed Lustre file system. Contents that match the prefix are added to a metadata record in the file system. When clients request a file, its contents are retrieved from the blob container and stored in the file system.

Use the Import prefix option on the Advanced tab to determine what data is imported from your blob container when the system is created. This field can't be changed after you create the Azure Managed Lustre file system.

  • The default import prefix, /, imports the entire contents of the blob container.

  • If you don't want to import files from the blob container, you can set an import prefix that doesn't match any files in the container.

  • If you use a hierarchical blob storage service (like NFSv3-mounted blob storage), you can think of the prefix as a file path. Items under the path are included in the Azure Managed Lustre file system.

  • If you use your blob container as a non-hierarchical object store, you can also think of the import prefix as a search string that is compared with the beginning of your blob object name. If the name of a file in your blob container starts with the string you specified as the import prefix, that file is made accessible in the file system. Lustre is a hierarchical file system, and / characters in blob file names become directory delimiters when stored in Lustre.

Metadata for exported files

When files are archived from the Azure Managed Lustre system to the blob container, additional metadata is saved to simplify reimporting the contents to an Azure Managed Lustre file system.

  • The following POSIX attributes from the Lustre file system are saved in the blob metadata as key-value pairs (value type in parentheses):

    • owner: (int)
    • group: (int)
    • permissions: (octal or rwxrwxrwx format; sticky bit is supported)

  • Directory attributes are saved in an empty blob. This blob has the same name as the directory path and contains the following key-value pair in the blob metadata:

    hdi_isfolder : true

You can modify the POSIX attributes manually before using the container to hydrate a new Lustre cluster. Edit or add blob metadata by using the key-value pairs described earlier.

Copy a Lustre blob container with AzCopy or Storage Explorer

You can move or copy the blob container Lustre uses by using AzCopy or Storage Explorer.

For AzCopy, you can include directory attributes by adding the following flag:

--include-directory-stub

Including this flag preserves directory POSIX attributes during a transfer, for example, owner, group, and permissions. If you use azcopy on the storage container without this flag, or with the flag set to false, then the data and directories are included in the transfer, but the directories don't retain their POSIX attributes.

In Storage Explorer, you can enable this flag in Settings by selecting Transfers and checking the box for Include Directory Stubs.

Screenshot showing how to include directory stubs during a transfer in Storage Explorer.

Next steps