az ml datastore

  • Reference

Note

This reference is part of the ml extension for the Azure CLI (version 2.15.0 or higher). The extension will automatically install the first time you run an az ml datastore command. Learn more about extensions.

Manage Azure ML datastores.

Azure ML datastores securely link your Azure storage services to your workspace so that you can access your storage without having to hardcode the connection information into your scripts. The connection secrets, like the storage service's authentication credentials, are stored in your workspace's Key Vault.

When you create a workspace, an Azure Storage account is automatically created as an associated resource. A blob container is created in this account, and its connection information is stored as a datastore named 'workspaceblobstore'. This serves as the workspace's default datastore, and the blob container is used to store your workspace artifacts and machine learning job logs and outputs.

Commands

Name Description Type Status
az ml datastore create

Create a datastore.

 Extension GA
az ml datastore delete

Delete a datastore.

 Extension GA
az ml datastore list

List datastores in a workspace.

 Extension GA
az ml datastore mount

Mount a specific datastore to a local path. For now only Linux is supported.

 Extension Preview
az ml datastore show

Show details for a datastore.

 Extension GA
az ml datastore update

Update a datastore.

 Extension GA

az ml datastore create

Create a datastore.

This connects the underlying Azure storage service to the workspace. The storage service types that can currently be connected to by creating a datastore include Azure Blob storage, Azure File Share, Azure Data Lake Storage Gen1 and Azure Data Lake Storage Gen2.

az ml datastore create --file
                       --resource-group
                       --workspace-name
                       [--name]
                       [--set]

Examples

Create a datastore from a YAML specification file

az ml datastore create --file blobstore.yml --resource-group my-resource-group --workspace-name my-workspace

Required Parameters

--file -f

Local path to the YAML file containing the Azure ML datastore specification. The YAML reference docs for datastore can be found at: https://aka.ms/ml-cli-v2-datastore-blob-yaml-reference, https://aka.ms/ml-cli-v2-datastore-file-yaml-reference, https://aka.ms/ml-cli-v2-datastore-data-lake-gen1-yaml-reference, https://aka.ms/ml-cli-v2-datastore-data-lake-gen2-yaml-reference.

--resource-group -g

Name of resource group. You can configure the default group using az configure --defaults group=<name>.

--workspace-name -w

Name of the Azure ML workspace. You can configure the default workspace using az configure --defaults workspace=<name>.

Optional Parameters

--name -n

Name of the datastore. This overwrites the 'name' field in the YAML file provided to --file/-f.

--set

Update an object by specifying a property path and value to set. Example: --set property1.property2=.

Global Parameters
--debug

Increase logging verbosity to show all debug logs.

--help -h

Show this help message and exit.

--only-show-errors

Only show errors, suppressing warnings.

--output -o

Output format.

accepted values: json, jsonc, none, table, tsv, yaml, yamlc
default value: json
--query

JMESPath query string. See http://jmespath.org/ for more information and examples.

--subscription

Name or ID of subscription. You can configure the default subscription using az account set -s NAME_OR_ID.

--verbose

Increase logging verbosity. Use --debug for full debug logs.

az ml datastore delete

Delete a datastore.

This deletes the connection information to the storage service from the workspace but does not delete the underlying data in storage.

az ml datastore delete --name
                       --resource-group
                       --workspace-name

Required Parameters

--name -n

Name of the datastore.

--resource-group -g

Name of resource group. You can configure the default group using az configure --defaults group=<name>.

--workspace-name -w

Name of the Azure ML workspace. You can configure the default workspace using az configure --defaults workspace=<name>.

Global Parameters
--debug

Increase logging verbosity to show all debug logs.

--help -h

Show this help message and exit.

--only-show-errors

Only show errors, suppressing warnings.

--output -o

Output format.

accepted values: json, jsonc, none, table, tsv, yaml, yamlc
default value: json
--query

JMESPath query string. See http://jmespath.org/ for more information and examples.

--subscription

Name or ID of subscription. You can configure the default subscription using az account set -s NAME_OR_ID.

--verbose

Increase logging verbosity. Use --debug for full debug logs.

az ml datastore list

List datastores in a workspace.

az ml datastore list --resource-group
                     --workspace-name
                     [--max-results]

Examples

List all the datastores in a workspace using --query argument to execute a JMESPath query on the results of commands.

az ml datastore list --query "[].{Name:name}"  --output table --resource-group my-resource-group --workspace-name my-workspace

Required Parameters

--resource-group -g

Name of resource group. You can configure the default group using az configure --defaults group=<name>.

--workspace-name -w

Name of the Azure ML workspace. You can configure the default workspace using az configure --defaults workspace=<name>.

Optional Parameters

--max-results -r

Max number of results to return.

Global Parameters
--debug

Increase logging verbosity to show all debug logs.

--help -h

Show this help message and exit.

--only-show-errors

Only show errors, suppressing warnings.

--output -o

Output format.

accepted values: json, jsonc, none, table, tsv, yaml, yamlc
default value: json
--query

JMESPath query string. See http://jmespath.org/ for more information and examples.

--subscription

Name or ID of subscription. You can configure the default subscription using az account set -s NAME_OR_ID.

--verbose

Increase logging verbosity. Use --debug for full debug logs.

az ml datastore mount

Preview

This command is in preview and under development. Reference and support levels: https://aka.ms/CLI_refstatus

Mount a specific datastore to a local path. For now only Linux is supported.

az ml datastore mount --path
                      [--mode]
                      [--mount-point]
                      [--persistent]
                      [--resource-group]
                      [--workspace-name]

Examples

Mount a datastore by name

az ml datastore mount --mount-point /mnt/my-datastore --mode ro_mount --path my-datastore

Mount a datastore by datastore short-form URL

az ml datastore mount --mount-point /mnt/my-datastore --mode ro_mount --path azureml://datastores/my-datastore

Mount a datastore by datastore long-form URL

az ml datastore mount --mount-point /mnt/my-datastore --mode ro_mount --path azureml://subscriptions/my-sub-id/resourcegroups/my-rg/providers/Microsoft.MachineLearningServices/workspaces/myworkspace/datastores/my-datastore

Required Parameters

--path

The datastore path to mount, in the form of <datastore_name> or azureml://datastores/<datastore_name>.

Optional Parameters

--mode

Mount mode, either ro_mount (read-only) or rw_mount (read-write).

default value: ro_mount
--mount-point

A local path used as mount point.

default value: /home/azureuser/mount/data
--persistent

Make mount persist across reboots. Supported only on Compute Instance.

default value: False
--resource-group -g

Name of resource group. You can configure the default group using az configure --defaults group=<name>.

--workspace-name -w

Name of the Azure ML workspace. You can configure the default workspace using az configure --defaults workspace=<name>.

Global Parameters
--debug

Increase logging verbosity to show all debug logs.

--help -h

Show this help message and exit.

--only-show-errors

Only show errors, suppressing warnings.

--output -o

Output format.

accepted values: json, jsonc, none, table, tsv, yaml, yamlc
default value: json
--query

JMESPath query string. See http://jmespath.org/ for more information and examples.

--subscription

Name or ID of subscription. You can configure the default subscription using az account set -s NAME_OR_ID.

--verbose

Increase logging verbosity. Use --debug for full debug logs.

az ml datastore show

Show details for a datastore.

az ml datastore show --name
                     --resource-group
                     --workspace-name

Required Parameters

--name -n

Name of the datastore.

--resource-group -g

Name of resource group. You can configure the default group using az configure --defaults group=<name>.

--workspace-name -w

Name of the Azure ML workspace. You can configure the default workspace using az configure --defaults workspace=<name>.

Global Parameters
--debug

Increase logging verbosity to show all debug logs.

--help -h

Show this help message and exit.

--only-show-errors

Only show errors, suppressing warnings.

--output -o

Output format.

accepted values: json, jsonc, none, table, tsv, yaml, yamlc
default value: json
--query

JMESPath query string. See http://jmespath.org/ for more information and examples.

--subscription

Name or ID of subscription. You can configure the default subscription using az account set -s NAME_OR_ID.

--verbose

Increase logging verbosity. Use --debug for full debug logs.

az ml datastore update

Update a datastore.

The 'description', 'tags', and 'credential' properties can be updated.

az ml datastore update --resource-group
                       --workspace-name
                       [--add]
                       [--file]
                       [--force-string]
                       [--name]
                       [--remove]
                       [--set]

Required Parameters

--resource-group -g

Name of resource group. You can configure the default group using az configure --defaults group=<name>.

--workspace-name -w

Name of the Azure ML workspace. You can configure the default workspace using az configure --defaults workspace=<name>.

Optional Parameters

--add

Add an object to a list of objects by specifying a path and key value pairs. Example: --add property.listProperty <key=value, string or JSON string>.

default value: []
--file -f

Local path to the YAML file containing the Azure ML datastore specification. The YAML reference docs for datastore can be found at: https://aka.ms/ml-cli-v2-datastore-blob-yaml-reference, https://aka.ms/ml-cli-v2-datastore-file-yaml-reference, https://aka.ms/ml-cli-v2-datastore-data-lake-gen1-yaml-reference, https://aka.ms/ml-cli-v2-datastore-data-lake-gen2-yaml-reference.

--force-string

When using 'set' or 'add', preserve string literals instead of attempting to convert to JSON.

default value: False
--name -n

Name of the datastore. This overwrites the 'name' field in the YAML file provided to --file/-f.

--remove

Remove a property or an element from a list. Example: --remove property.list <indexToRemove> OR --remove propertyToRemove.

default value: []
--set

Update an object by specifying a property path and value to set. Example: --set property1.property2=<value>.

default value: []
Global Parameters
--debug

Increase logging verbosity to show all debug logs.

--help -h

Show this help message and exit.

--only-show-errors

Only show errors, suppressing warnings.

--output -o

Output format.

accepted values: json, jsonc, none, table, tsv, yaml, yamlc
default value: json
--query

JMESPath query string. See http://jmespath.org/ for more information and examples.

--subscription

Name or ID of subscription. You can configure the default subscription using az account set -s NAME_OR_ID.

--verbose

Increase logging verbosity. Use --debug for full debug logs.