Azure Managed Lustre Auto-Export (Preview)

The Auto-Export feature for Azure Managed Lustre is a capability that automatically synchronizes changes in your Azure Managed Lustre file system with a linked Azure Blob Storage Container. This feature ensures that new and modified files in the file system are reflected in the associated Blob Storage Container without manual intervention. This streamlines data management and improves synchronization for long-term storage.

How auto-export works

Auto-Export operates by continuously monitoring changes in the Azure Managed Lustre file system. Based on the configured export policy, it updates the contents of the associated Blob Storage Container to reflect these changes. This feature provides users with a seamless and automated data replication process.

Configuration

Auto-Export is enabled on an existing Azure Managed Lustre file system that has an associated Blob Storage Container configured. Auto-Export is configured in the Blob Integration settings in the Azure portal.

To create a new Auto-Export job, follow these steps:

1. In the Azure portal, open your Azure Managed Lustre file system and select the Blob integration pane under Settings.
2. Select + Create new job.
3. In the Job Type dropdown, select Auto-Export.
4. In the File system path field, you can enter a string to specify a directory path. All new or changed files in the file system under this path (prefix) are continually exported. Files are written to the blob container with the same file path (or prefix) that they have in the Lustre system. To avoid overwriting existing files in the blob container, ensure that the path of the file in your Lustre system doesn't overlap with the existing path of the file in the blob container unless your Lustre file was imported from that path in the Blob Container.

Auto-Export Behavior

Auto-Export handles different types of changes as follows:

• New File Creation, New Directory, and File Content Changes: Blob Integration identifies new files and directories and changed data and starts an export job automatically. Auto-Export ensures that the latest version of the file is transferred to Blob Storage.
• Metadata Changes: Changes such as renames, ownership updates, or permission adjustments are not synchronized currently.
• Deletion: When files, directories, or symlinks are deleted in the file system, they are not removed from the Blob Storage Container.

No more than one Blob Integration job (Manual Export, Auto Export, Import) can be run at a time. Auto-Export is implemented as continuous export iterations. Upon completion of an iteration, Blob Integration scans the filesystem for any new files, directories, or content changes and starts a new iteration of an export job. Logs created in your configured Logging container can help identify synchronization issues and understand the reasons behind failed operations.

Monitoring and Managing Auto-Export

You can monitor Auto-Export activities and manage configurations using the Azure portal:

The Blob Integration pane displays details of export activities in the Recent jobs section, including the status of recent jobs and metrics related to automatic synchronization.

To cancel the job that's in progress, select the Cancel link for that job in the Recent jobs table. The Cancel link is only available for the current auto-export.

To view the metrics of an Auto-Export job, click on the Name of the job, and the Metrics blade appears on the side panel in the Portal.

Auto-Export Job Monitoring

The Blob Integration pane displays details of export activities in the Recent jobs section, including the status of recent jobs and metrics related to automatic synchronization. To cancel the job that's in progress, select the Cancel link for that job in the Recent jobs table. The Cancel link is only available for the current auto-export. To view the metrics of an Auto-Export job, click on the Name of the job, and the Metrics blade appears on the side panel in the Portal.

Metrics

Metrics are grouped into two main categories, Overall and Current Iteration.

Overall Statistics

Statistics since enabling Auto-Export

Total Files Exported

Count of files successfully copied to the associated Blob Container since enabling Auto-Export

Total MiB Exported

Aggregate file size (in MiB) successfully copied to the associated Blob Container since enabling Auto-Export

Iterations Completed

Count of times Blob Integration has identified new or changed data and initiated a new Export job

Last Successful Iteration

Finish timestamp of the last iteration that successfully exported all data to the associated Blob Container at the time

Current Iteration Statistics

Statistics about the current activity

Files Discovered

Count of files that the Export job is currently exporting, including files already successfully copied to the associated Blob Container in this Iteration*

MiB Discovered

Aggregate file size (in MiB) that the Export job is currently exporting, including files already successfully copied to the associated Blob Container in this Iteration*

Files Exported

Count of files of new and changed data successfully copied to the associated Blob Container

MiB Exported

Aggregate file size (in MiB) of new and changed data successfully copied to the associated Blob Container

Files Failed

Total number of files that failed to copy during the current iteration. Click on this link to be taken to the Logging Container page to view the logs associated with this Auto-Export job.

* The metrics for the current ongoing iteration should be read as “so far.” For example, Files Discovered is the number of the files discovered for exporting at the moment of reporting. The next stats reporting cycle may show that more files are discovered.

Considerations and Best Practices

While using Auto-Export, consider the following best practices to ensure smooth operation:

• Conflict Management: If a file is modified in both the file system and Blob Storage, there is a risk of overwrite. Use application-level coordination to prevent conflicting edits.
• Disable Auto-Export Before Deletion: Before deleting a file system or its Blob Integration, ensure that all queued updates are synchronized. Verify that the Last Successful Iteration Time is recent and the Files Failed metric is zero to avoid data loss. Then, disable Auto-Export.

Next step

• Learn more about Azure Blob Storage integration with Azure Managed Lustre file systems.