AzureFileSystemProvider Class
- java.
lang. Object - java.
nio. file. spi. FileSystemProvider - com.
azure. storage. blob. nio. AzureFileSystemProvider
- com.
- java.
public final class AzureFileSystemProvider
extends FileSystemProvider
The AzureFileSystemProvider is Azure Storage's implementation of the nio interface on top of Azure Blob Storage.
Particular care should be taken when working with a remote storage service. This implementation makes no guarantees on behavior or state should other processes operate on the same data concurrently; file systems from this provider will assume they have exclusive access to their data and will behave without regard for potential of interfering applications. Moreover, remote file stores introduce higher latencies. Therefore, additional consideration should be given to managing concurrency: race conditions are more likely to manifest and network failures occur more frequently than disk failures. These and other such distributed application scenarios must be considered when working with this file system. While the AzureFileSystem will ensure it takes appropriate steps towards robustness and reliability, the application developer must design around these failure scenarios and have fallback and retry options available.
The Azure Blob Storage service backing these APIs is not a true FileSystem, nor is it the goal of this implementation to force Azure Blob Storage to act like a full-fledged file system. Some APIs and scenarios will remain unsupported indefinitely until they may be sensibly implemented. Other APIs may experience lower performance than is expected because of the number of network requests needed to ensure correctness. The javadocs for each type and method should also be read carefully to understand what guarantees are made and how they may differ from the contract defined by FileSystemProvider.
The scheme for this provider is "azb", and the format of the URI to identify an AzureFileSystem is "azb://?endpoint=". The endpoint of the Storage account is used to uniquely identify the filesystem.
An AzureFileSystem is backed by an account. An AzureFileStore is backed by a container. Any number of containers may be specified as file stores upon creation of the file system. When a file system is created, it will try to retrieve the properties of each container to ensure connection to the account. If any of the containers does not exist, it will be created. Failure to access or create containers as necessary will result in an exception and failure to create the file system. Any data existing in the containers will be preserved and accessible via the file system, though customers should be aware that it must be in a format understandable by the types in this package or behavior will be undefined.
newFileSystem(URI uri, Map<String,?> config) will check for the following keys in the configuration map and expect the named types. Any entries not listed here will be ignored. Note that AzureFileSystem has public constants defined for each of the keys for convenience. Most values are documented in the blob package. Any values which are unique to nio will be documented here.
AzureStorageSharedKeyCredential:com.azure.storage.common.StorageSharedKeyCredentialAzureStorageSasTokenCredential:com.azure.core.credential.AzureSasCredentialAzureStorageHttpLogDetailLevel:com.azure.core.http.policy.HttpLogDetailLevelAzureStorageMaxTries:IntegerAzureStorageTryTimeout:IntegerAzureStorageRetryDelayInMs:LongAzureStorageMaxRetryDelayInMs:LongAzureStorageRetryPolicyType:com.azure.storage.common.policy.RetryPolicyTypeAzureStorageSecondaryHost:StringAzureStorageSecondaryHost:IntegerAzureStorageBlockSize:LongAzureStoragePutBlobThreshold:LongAzureStorageMaxConcurrencyPerRequest:IntegerAzureStorageDownloadResumeRetries:IntegerAzureStorageFileStores:StringAzureStorageSkipInitialContainerCheck:Boolean. Indicates that the initial check which confirms the existence of the containers meant to act as file stores should be skipped. This can be useful in cases where a sas token that is scoped to only one file is used to authenticate.
Either an account key or a sas token must be specified. If both are provided, the account key will be preferred. If a sas token is specified, the customer must take care that it has appropriate permissions to perform the actions demanded of the file system in a given workflow, including the initial connection check specified above. The same token will be applied to all operations.
An iterable of file stores must also be provided; each entry should simply be the name of a container. The first container listed will be considered the default file store and the root directory of which will be the file system's default directory. All other values listed are used to configure the underlying com.azure.storage.blob.BlobServiceClient. Please refer to that type for more information on these values.
Field Summary
| Modifier and Type | Field and Description |
|---|---|
| static final String |
CACHE_CONTROL
A helper for setting the HTTP properties when creating a directory. |
| static final String |
CONTENT_DISPOSITION
A helper for setting the HTTP properties when creating a directory. |
| static final String |
CONTENT_ENCODING
A helper for setting the HTTP properties when creating a directory. |
| static final String |
CONTENT_LANGUAGE
A helper for setting the HTTP properties when creating a directory. |
| static final String |
CONTENT_MD5
A helper for setting the HTTP properties when creating a directory. |
| static final String |
CONTENT_TYPE
A helper for setting the HTTP properties when creating a directory. |
Constructor Summary
| Constructor | Description |
|---|---|
| AzureFileSystemProvider() |
Creates an Azure |
Method Summary
Methods inherited from java.lang.Object
Methods inherited from java.nio.file.spi.FileSystemProvider
Field Details
CACHE_CONTROL
public static final String CACHE_CONTROL
A helper for setting the HTTP properties when creating a directory.
CONTENT_DISPOSITION
public static final String CONTENT_DISPOSITION
A helper for setting the HTTP properties when creating a directory.
CONTENT_ENCODING
public static final String CONTENT_ENCODING
A helper for setting the HTTP properties when creating a directory.
CONTENT_LANGUAGE
public static final String CONTENT_LANGUAGE
A helper for setting the HTTP properties when creating a directory.
CONTENT_MD5
public static final String CONTENT_MD5
A helper for setting the HTTP properties when creating a directory.
CONTENT_TYPE
public static final String CONTENT_TYPE
A helper for setting the HTTP properties when creating a directory.
Constructor Details
AzureFileSystemProvider
public AzureFileSystemProvider()
Creates an AzureFileSystemProvider.
Method Details
checkAccess
public void checkAccess(Path path, AccessMode[] accessModes)
Checks the existence, and optionally the accessibility, of a file.
This method may only be used to check the existence of a file. It is not possible to determine the permissions granted to a given client, so if any mode argument is specified, an UnsupportedOperationException will be thrown.
Overrides:
AzureFileSystemProvider.checkAccess(Path path, AccessMode[] accessModes)Parameters:
Throws:
copy
public void copy(Path source, Path destination, CopyOption[] copyOptions)
Copies the resource at the source location to the destination.
This method is not atomic with respect to other file system operations. More specifically, the checks necessary to validate the inputs and state of the file system are not atomic with the actual copying of data. If the copy is triggered, the copy itself is atomic and only a complete copy will ever be left at the destination.
In addition to those in the docs for FileSystemProvider#copy(Path, Path, CopyOption...), this method has the following requirements for successful completion. StandardCopyOption#COPY_ATTRIBUTES must be passed as it is impossible not to copy blob properties; if this option is not passed, an UnsupportedOperationException will be thrown. Neither the source nor the destination can be a root directory; if either is a root directory, an IllegalArgumentException will be thrown. The parent directory of the destination must at least weakly exist; if it does not, an IOException will be thrown. The only supported option other than StandardCopyOption#COPY_ATTRIBUTES is StandardCopyOption#REPLACE_EXISTING; the presence of any other option will result in an UnsupportedOperationException.
This method supports both virtual and concrete directories as both the source and destination. Unlike when creating a directory, the existence of a virtual directory at the destination will cause this operation to fail. This is in order to prevent the possibility of overwriting a non-empty virtual directory with a file. Still, as mentioned above, this check is not atomic with the creation of the resultant directory.
Overrides:
AzureFileSystemProvider.copy(Path source, Path destination, CopyOption[] copyOptions)Parameters:
Throws:
createDirectory
public void createDirectory(Path path, FileAttribute<?>[] fileAttributes)
Creates a new directory at the specified path.
The existence of a directory in the AzureFileSystem is defined on two levels. Weak existence is defined by the presence of a non-zero number of blobs prefixed with the directory's path. This concept is also known as a virtual directory and enables the file system to work with containers that were pre-loaded with data by another source but need to be accessed by this file system. Strong existence is defined as the presence of an actual storage resource at the given path, which in the case of directories, is a zero-length blob whose name is the directory path with a particular metadata field indicating the blob's status as a directory. This is also known as a concrete directory. Directories created by this file system will strongly exist. Operations targeting directories themselves as the object (e.g. setting properties) will target marker blobs underlying concrete directories. Other operations (e.g. listing) will operate on the blob-name prefix.
This method fulfills the nio contract of: "The check for the existence of the file and the creation of the directory if it does not exist are a single operation that is atomic with respect to all other filesystem activities that might affect the directory." More specifically, this method will atomically check for strong existence of another file or directory at the given path and fail if one is present. On the other hand, we only check for weak existence of the parent to determine if the given path is valid. Additionally, the action of checking whether the parent exists, is not atomic with the creation of the directory. Note that while it is possible that the parent may be deleted between when the parent is determined to exist and the creation of the child, the creation of the child will always ensure the existence of a virtual parent, so the child will never be left floating and unreachable. The different checks on parent and child is due to limitations in the Storage service API.
There may be some unintuitive behavior when working with directories in this file system, particularly virtual directories (usually those not created by this file system). A virtual directory will disappear as soon as all its children have been deleted. Furthermore, if a directory with the given path weakly exists at the time of calling this method, this method will still return success and create a concrete directory at the target location. In other words, it is possible to "double create" a directory if it first weakly exists and then is strongly created. This is both because it is impossible to atomically check if a virtual directory exists while creating a concrete directory and because such behavior will have minimal side effects--no files will be overwritten and the directory will still be available for writing as intended, though it may not be empty. This is not a complete list of such unintuitive behavior.
This method will attempt to extract standard HTTP content headers from the list of file attributes to set them as blob headers. All other attributes will be set as blob metadata. The value of every attribute will be converted to a String except the Content-MD5 attribute which expects a byte[]. When extracting the content headers, the following strings will be used for comparison (constants for these values can be found on this type):
Content-TypeContent-DispositionContent-LanguageContent-EncodingContent-MD5Cache-Control
Note that these properties also have a particular semantic in that if one is specified, all are updated. In other words, if any of the above is set, all those that are not set will be cleared. See the Azure Docs for more information.
Overrides:
AzureFileSystemProvider.createDirectory(Path path, FileAttribute<?>[] fileAttributes)Parameters:
Throws:
delete
public void delete(Path path)
Deletes the specified resource.
This method is not atomic with respect to other file system operations. It is possible to delete a file in use by another process, and doing so will not immediately invalidate any channels open to that file--they will simply start to fail. Root directories cannot be deleted even when empty.
Overrides:
AzureFileSystemProvider.delete(Path path)Parameters:
Throws:
getFileAttributeView
public V <V>getFileAttributeView(Path path, Class<V> type, LinkOption[] linkOptions)
Returns a file attribute view of a given type.
See AzureBasicFileAttributeView and AzureBlobFileAttributeView for more information.
Reading attributes on a virtual directory will return null for most properties other than isVirtualDirectory(), which will return true. See createDirectory(Path path, FileAttribute<?>[] fileAttributes) for more information on virtual directories.
Overrides:
AzureFileSystemProvider.getFileAttributeView(Path path, Class<V> type, LinkOption[] linkOptions)Parameters:
Returns:
getFileStore
public FileStore getFileStore(Path path)
Unsupported.
Overrides:
AzureFileSystemProvider.getFileStore(Path path)Parameters:
Returns:
Throws:
getFileSystem
public FileSystem getFileSystem(URI uri)
Returns an existing FileSystem created by this provider.
The format of a URI identifying a file system is "azb://?endpoint=<endpoint>".
Trying to retrieve a closed file system will throw a FileSystemNotFoundException. Once closed, a file system with the same identifier may be reopened.
Overrides:
AzureFileSystemProvider.getFileSystem(URI uri)Parameters:
Returns:
getPath
public Path getPath(URI uri)
Return a Path object by converting the given URI. The resulting Path is associated with a FileSystem that already exists.
Overrides:
AzureFileSystemProvider.getPath(URI uri)Parameters:
Returns:
getScheme
public String getScheme()
Returns the URI scheme that identifies this provider: "azb".
Overrides:
AzureFileSystemProvider.getScheme()Returns:
"azb"isHidden
public boolean isHidden(Path path)
Always returns false as hidden files are not supported.
Overrides:
AzureFileSystemProvider.isHidden(Path path)Parameters:
Returns:
Throws:
isSameFile
public boolean isSameFile(Path path, Path path1)
Unsupported.
Overrides:
AzureFileSystemProvider.isSameFile(Path path, Path path1)Parameters:
Throws:
move
public void move(Path path, Path path1, CopyOption[] copyOptions)
Unsupported.
Overrides:
AzureFileSystemProvider.move(Path path, Path path1, CopyOption[] copyOptions)Parameters:
Throws:
newByteChannel
public SeekableByteChannel newByteChannel(Path path, Set set, FileAttribute[] fileAttributes)
Opens or creates a file, returning a seekable byte channel to access the file.
This method is primarily offered to support some jdk convenience methods such as Files#createFile(Path, FileAttribute[]) which requires opening a channel and closing it. A channel may only be opened in read mode OR write mode. It may not be opened in read/write mode. Seeking is supported for reads, but not for writes. Modifications to existing files is not permitted--only creating new files or overwriting existing files.
This type is not threadsafe to prevent having to hold locks across network calls.
Overrides:
AzureFileSystemProvider.newByteChannel(Path path, Set<? extends OpenOption> set, FileAttribute<?>[] fileAttributes)Parameters:
Returns:
Throws:
newDirectoryStream
public DirectoryStream<Path> newDirectoryStream(Path path, DirectoryStream.Filter<? super Path> filter)
Returns an AzureDirectoryStream for iterating over the contents of a directory. The elements returned by the directory stream's iterator are of type Path, each one representing an entry in the directory. The Path objects are obtained as if by resolving the name of the directory entry against dir. The entries returned by the iterator are filtered by the given filter.
When not using the try-with-resources construct, then directory stream's close method should be invoked after iteration is completed to free any resources held for the open directory.
Where the filter terminates due to an uncaught error or runtime exception then it is propagated to the hasNext or next method. Where an IOException is thrown, it results in the hasNext or next method throwing a DirectoryIteratorException with the IOException as the cause.
Overrides:
AzureFileSystemProvider.newDirectoryStream(Path path, DirectoryStream.Filter<? super Path> filter)Parameters:
Returns:
DirectoryStream objectThrows:
newFileSystem
public FileSystem newFileSystem(URI uri, Map<String,?> config)
Constructs a new FileSystem object identified by a URI.
The format of a URI identifying a file system is "azb://?endpoint=".
Once closed, a file system with the same identifier may be reopened.
Overrides:
AzureFileSystemProvider.newFileSystem(URI uri, Map<String,?> config)Parameters:
Returns:
Throws:
newInputStream
public InputStream newInputStream(Path path, OpenOption[] options)
Opens an InputStream to the given path.
The stream will not attempt to read or buffer the entire file. However, when fetching data, it will always request the same size chunk of several MB to prevent network thrashing on small reads. Mark and reset are supported.
Only StandardOpenOption#READ is supported. Any other option will throw.
Overrides:
AzureFileSystemProvider.newInputStream(Path path, OpenOption[] options)Parameters:
Returns:
Throws:
newOutputStream
public OutputStream newOutputStream(Path path, OpenOption[] options)
Opens an OutputStream to the given path. The resulting file will be stored as a block blob.
The only supported options are StandardOpenOption#CREATE, StandardOpenOption#CREATE_NEW, StandardOpenOption#WRITE, StandardOpenOption#TRUNCATE_EXISTING. Any other options will throw an UnsupportedOperationException. WRITE and TRUNCATE_EXISTING must be specified or an IllegalArgumentException will be thrown. Hence, files cannot be updated, only overwritten completely.
This stream will not attempt to buffer the entire file, however some buffering will be done for potential optimizations and to avoid network thrashing. Specifically, up to AZURE_STORAGE_PUT_BLOB_THRESHOLD bytes will be buffered initially. If that threshold is exceeded, the data will be broken into chunks and sent in blocks, and writes will be buffered into sizes of AZURE_STORAGE_UPLOAD_BLOCK_SIZE. The maximum number of buffers of this size to be allocated is defined by AZURE_STORAGE_MAX_CONCURRENCY_PER_REQUEST, which also configures the level of parallelism with which we may write and thus may affect write speeds as well.
The data is only committed when the steam is closed. Hence, data cannot be read from the destination until the stream is closed. When the close method returns, it is guaranteed that, barring any errors, the data is finalized and available for reading.
Writing happens asynchronously. Bytes passed for writing are stored until either the threshold or block size are met at which time they are sent to the service. When the write method returns, there is no guarantee about which phase of this process the data is in other than it has been accepted and will be written. Again, closing will guarantee that the data is written and available.
Flush is a no-op as regards data transfers, but it can be used to check the state of the stream for errors. This can be a useful tool because writing happens asynchronously, and therefore an error from a previous write may not otherwise be thrown unless the stream is flushed, closed, or written to again.
Overrides:
AzureFileSystemProvider.newOutputStream(Path path, OpenOption[] options)Parameters:
Returns:
Throws:
readAttributes
public A readAttributes(Path path, Class type, LinkOption[] linkOptions)
Reads a file's attributes as a bulk operation.
See AzureBasicFileAttributes and AzureBlobFileAttributes for more information.
Reading attributes on a virtual directory will return null for most properties other than isVirtualDirectory(), which will return true. See createDirectory(Path path, FileAttribute<?>[] fileAttributes) for more information on virtual directories.
Overrides:
AzureFileSystemProvider.readAttributes(Path path, Class<A> type, LinkOption[] linkOptions)Parameters:
Returns:
Throws:
readAttributes
public Map<String,Object> readAttributes(Path path, String attributes, LinkOption[] linkOptions)
Reads a set of file attributes as a bulk operation.
See AzureBasicFileAttributes and AzureBlobFileAttributes for more information.
Reading attributes on a virtual directory will return null for all properties other than isVirtualDirectory(), which will return true. See createDirectory(Path path, FileAttribute<?>[] fileAttributes) for more information on virtual directories.
Overrides:
AzureFileSystemProvider.readAttributes(Path path, String attributes, LinkOption[] linkOptions)Parameters:
Returns:
Throws:
setAttribute
public void setAttribute(Path path, String attributes, Object value, LinkOption[] linkOptions)
Sets the value of a file attribute.
See AzureBlobFileAttributeView for more information.
Setting attributes on a virtual directory is not supported and will throw an IOException. See createDirectory(Path path, FileAttribute<?>[] fileAttributes) for more information on virtual directories.
Overrides:
AzureFileSystemProvider.setAttribute(Path path, String attributes, Object value, LinkOption[] linkOptions)Parameters:
Throws:
