Share via


az ml job

Note

This reference is part of the ml extension for the Azure CLI (version 2.15.0 or higher). The extension will automatically install the first time you run an az ml job command. Learn more about extensions.

Manage Azure ML jobs.

An Azure ML job executes a task against a specified compute target. You can configure jobs to scale out model training on Azure. Azure ML supports different job types with different capabilities. For example, the most basic job, a command job, executes a command in a Docker container and can be leveraged for single-node and distributed training. A sweep job executes a hyperparameter sweep over a specified search space for tuning a model's hyperparameters.

Jobs also enable systematic tracking for your ML experimentation and workflows. Once a job is created, Azure ML maintains a run record for the job that includes the metadata, any metrics, logs, and artifacts generated during the job, code that was executed, and the Azure ML environment used. All of your jobs' run records can be viewed in Azure ML studio.

Commands

Name Description Type Status
az ml job archive

Archive a job.

Extension GA
az ml job cancel

Cancel a job.

Extension GA
az ml job connect-ssh

Set up ssh connection and sends the request to the SSH service running inside user's container through Tundra.

Extension GA
az ml job create

Create a job.

Extension GA
az ml job download

Download all job-related files.

Extension GA
az ml job list

List jobs in a workspace.

Extension GA
az ml job restore

Restore an archived job.

Extension GA
az ml job show

Show details for a job.

Extension GA
az ml job show-services

Show services of a job per node.

Extension GA
az ml job stream

Stream job logs to the console.

Extension GA
az ml job update

Update a job.

Extension GA
az ml job validate

Validate a job. This command works for pipeline jobs only for now.

Extension GA

az ml job archive

Archive a job.

Archiving a job will hide it by default from list queries (az ml job list). You can still continue to reference and use an archived job in your workflows. Only completed jobs can be archived.

az ml job archive --name
                  --resource-group
                  --workspace-name

Required Parameters

--name -n

Name of the job.

--resource-group -g

Name of resource group. You can configure the default group using az configure --defaults group=<name>.

--workspace-name -w

Name of the Azure ML workspace. You can configure the default workspace using az configure --defaults workspace=<name>.

Global Parameters
--debug

Increase logging verbosity to show all debug logs.

--help -h

Show this help message and exit.

--only-show-errors

Only show errors, suppressing warnings.

--output -o

Output format.

Accepted values: json, jsonc, none, table, tsv, yaml, yamlc
Default value: json
--query

JMESPath query string. See http://jmespath.org/ for more information and examples.

--subscription

Name or ID of subscription. You can configure the default subscription using az account set -s NAME_OR_ID.

--verbose

Increase logging verbosity. Use --debug for full debug logs.

az ml job cancel

Cancel a job.

az ml job cancel --name
                 --resource-group
                 --workspace-name

Examples

Cancel a job by name

az ml job cancel --name my-job-id --resource-group my-resource-group --workspace-name my-workspace

Required Parameters

--name -n

Name of the job.

--resource-group -g

Name of resource group. You can configure the default group using az configure --defaults group=<name>.

--workspace-name -w

Name of the Azure ML workspace. You can configure the default workspace using az configure --defaults workspace=<name>.

Global Parameters
--debug

Increase logging verbosity to show all debug logs.

--help -h

Show this help message and exit.

--only-show-errors

Only show errors, suppressing warnings.

--output -o

Output format.

Accepted values: json, jsonc, none, table, tsv, yaml, yamlc
Default value: json
--query

JMESPath query string. See http://jmespath.org/ for more information and examples.

--subscription

Name or ID of subscription. You can configure the default subscription using az account set -s NAME_OR_ID.

--verbose

Increase logging verbosity. Use --debug for full debug logs.

az ml job connect-ssh

Set up ssh connection and sends the request to the SSH service running inside user's container through Tundra.

az ml job connect-ssh --name
                      --resource-group
                      --workspace-name
                      [--node-index]
                      [--private-key-file-path]

Examples

Set up ssh connection and sends the request to the SSH service.

az ml job connect-ssh --name my-job-id --node-index 0 --private-key-file-path "C:/Temp/.ssh/id_rsa" --resource-group my-resource-group --workspace-name my-workspace

Required Parameters

--name -n

Name of the job.

--resource-group -g

Name of resource group. You can configure the default group using az configure --defaults group=<name>.

--workspace-name -w

Name of the Azure ML workspace. You can configure the default workspace using az configure --defaults workspace=<name>.

Optional Parameters

--node-index -i

The index of the node to connect through ssh.

Default value: 0
--private-key-file-path -f

The path to the private key file file.

Global Parameters
--debug

Increase logging verbosity to show all debug logs.

--help -h

Show this help message and exit.

--only-show-errors

Only show errors, suppressing warnings.

--output -o

Output format.

Accepted values: json, jsonc, none, table, tsv, yaml, yamlc
Default value: json
--query

JMESPath query string. See http://jmespath.org/ for more information and examples.

--subscription

Name or ID of subscription. You can configure the default subscription using az account set -s NAME_OR_ID.

--verbose

Increase logging verbosity. Use --debug for full debug logs.

az ml job create

Create a job.

To create a job, you will typically need to configure any code to be run, an environment encapsulating the dependencies, a compute target to execute the job on, and any additional job-specific settings. When a job is created, it is submitted for execution against the specified compute resource.

az ml job create --file
                 --resource-group
                 --workspace-name
                 [--name]
                 [--save-as]
                 [--set]
                 [--skip-validation]
                 [--stream]
                 [--web]

Examples

Create a job from a YAML specification file

az ml job create --file job.yml --resource-group my-resource-group --workspace-name my-workspace

Create a job from a YAML specification file and open the job's run details in the Azure ML studio portal

az ml job create --file job.yml --web --resource-group my-resource-group --workspace-name my-workspace

Required Parameters

--file -f

Local path to the YAML file containing the Azure ML job specification. The YAML reference docs for job can be found at: https://aka.ms/ml-cli-v2-job-command-yaml-reference, https://aka.ms/ml-cli-v2-job-sweep-yaml-reference, https://aka.ms/ml-cli-v2-job-pipeline-yaml-reference.

--resource-group -g

Name of resource group. You can configure the default group using az configure --defaults group=<name>.

--workspace-name -w

Name of the Azure ML workspace. You can configure the default workspace using az configure --defaults workspace=<name>.

Optional Parameters

--name -n

Name of the job.

--save-as -a

File to which the created job's state in YAML format will be written.

--set

Update an object by specifying a property path and value to set. Example: --set property1.property2=.

--skip-validation

Skip validation in creating the resource. Note that dependent resources will not skip their validation in creating.

Default value: False
--stream -s

Indicates whether to stream the job's logs to the console.

Default value: False
--web -e

Show the job's run details in Azure ML studio in a web browser.

Default value: False
Global Parameters
--debug

Increase logging verbosity to show all debug logs.

--help -h

Show this help message and exit.

--only-show-errors

Only show errors, suppressing warnings.

--output -o

Output format.

Accepted values: json, jsonc, none, table, tsv, yaml, yamlc
Default value: json
--query

JMESPath query string. See http://jmespath.org/ for more information and examples.

--subscription

Name or ID of subscription. You can configure the default subscription using az account set -s NAME_OR_ID.

--verbose

Increase logging verbosity. Use --debug for full debug logs.

az ml job download

Download all job-related files.

The files will be downloaded in a folder named after the job's name.

az ml job download --name
                   --resource-group
                   --workspace-name
                   [--all]
                   [--download-path]
                   [--output-name]

Examples

Download a job's logs and outputs to the current working directory

az ml job download --name my-job --resource-group my-resource-group --workspace-name my-workspace

Required Parameters

--name -n

Name of the job.

--resource-group -g

Name of resource group. You can configure the default group using az configure --defaults group=<name>.

--workspace-name -w

Name of the Azure ML workspace. You can configure the default workspace using az configure --defaults workspace=<name>.

Optional Parameters

--all

Download all the outputs of the job.

Default value: False
--download-path -p

Path to download the job files to. If omitted, job files will be downloaded to the current directory.

--output-name

The name of the user-defined output to download. This should correspond to a key in the outputs dictionary of a job. If omitted, the job's default artifact output files will be downloaded.

Global Parameters
--debug

Increase logging verbosity to show all debug logs.

--help -h

Show this help message and exit.

--only-show-errors

Only show errors, suppressing warnings.

--output -o

Output format.

Accepted values: json, jsonc, none, table, tsv, yaml, yamlc
Default value: json
--query

JMESPath query string. See http://jmespath.org/ for more information and examples.

--subscription

Name or ID of subscription. You can configure the default subscription using az account set -s NAME_OR_ID.

--verbose

Increase logging verbosity. Use --debug for full debug logs.

az ml job list

List jobs in a workspace.

az ml job list --resource-group
               --workspace-name
               [--all-results {false, true}]
               [--archived-only]
               [--include-archived]
               [--max-results]
               [--parent-job-name]

Examples

List all the jobs status in a workspace using --query argument to execute a JMESPath query on the results of commands.

az ml job list --query "[].{Name:name,Jobstatus:status}"  --output table --resource-group my-resource-group --workspace-name my-workspace

Required Parameters

--resource-group -g

Name of resource group. You can configure the default group using az configure --defaults group=<name>.

--workspace-name -w

Name of the Azure ML workspace. You can configure the default workspace using az configure --defaults workspace=<name>.

Optional Parameters

--all-results

Returns all results.

Accepted values: false, true
Default value: False
--archived-only

List archived jobs only.

Default value: False
--include-archived

List archived jobs and active jobs.

Default value: False
--max-results -r

Max number of results to return. Default is 50.

Default value: 50
--parent-job-name -p

Name of the parent job. Will list all jobs whose parent_job_name matches the given name.

Global Parameters
--debug

Increase logging verbosity to show all debug logs.

--help -h

Show this help message and exit.

--only-show-errors

Only show errors, suppressing warnings.

--output -o

Output format.

Accepted values: json, jsonc, none, table, tsv, yaml, yamlc
Default value: json
--query

JMESPath query string. See http://jmespath.org/ for more information and examples.

--subscription

Name or ID of subscription. You can configure the default subscription using az account set -s NAME_OR_ID.

--verbose

Increase logging verbosity. Use --debug for full debug logs.

az ml job restore

Restore an archived job.

When an archived job is restored, it will no longer be hidden from list queries (az ml job list).

az ml job restore --name
                  --resource-group
                  --workspace-name

Required Parameters

--name -n

Name of the job.

--resource-group -g

Name of resource group. You can configure the default group using az configure --defaults group=<name>.

--workspace-name -w

Name of the Azure ML workspace. You can configure the default workspace using az configure --defaults workspace=<name>.

Global Parameters
--debug

Increase logging verbosity to show all debug logs.

--help -h

Show this help message and exit.

--only-show-errors

Only show errors, suppressing warnings.

--output -o

Output format.

Accepted values: json, jsonc, none, table, tsv, yaml, yamlc
Default value: json
--query

JMESPath query string. See http://jmespath.org/ for more information and examples.

--subscription

Name or ID of subscription. You can configure the default subscription using az account set -s NAME_OR_ID.

--verbose

Increase logging verbosity. Use --debug for full debug logs.

az ml job show

Show details for a job.

az ml job show --name
               --resource-group
               --workspace-name
               [--web]

Examples

Show the status of a job using --query argument to execute a JMESPath query on the results of commands.

az ml job show --name my-job-id --query "{Name:name,Jobstatus:status}"  --output table --resource-group my-resource-group --workspace-name my-workspace

Required Parameters

--name -n

Name of the job.

--resource-group -g

Name of resource group. You can configure the default group using az configure --defaults group=<name>.

--workspace-name -w

Name of the Azure ML workspace. You can configure the default workspace using az configure --defaults workspace=<name>.

Optional Parameters

--web -e

Show the job's run details in Azure ML studio in a web browser.

Default value: False
Global Parameters
--debug

Increase logging verbosity to show all debug logs.

--help -h

Show this help message and exit.

--only-show-errors

Only show errors, suppressing warnings.

--output -o

Output format.

Accepted values: json, jsonc, none, table, tsv, yaml, yamlc
Default value: json
--query

JMESPath query string. See http://jmespath.org/ for more information and examples.

--subscription

Name or ID of subscription. You can configure the default subscription using az account set -s NAME_OR_ID.

--verbose

Increase logging verbosity. Use --debug for full debug logs.

az ml job show-services

Show services of a job per node.

az ml job show-services --name
                        --resource-group
                        --workspace-name
                        [--node-index]

Examples

Show the services of a job per node using --query argument to execute a JMESPath query on the results of commands.

az ml job show-services --name my-job-id --node-index 0 --query "{Name:name,Jobstatus:status}"  --output table --resource-group my-resource-group --workspace-name my-workspace

Required Parameters

--name -n

Name of the job.

--resource-group -g

Name of resource group. You can configure the default group using az configure --defaults group=<name>.

--workspace-name -w

Name of the Azure ML workspace. You can configure the default workspace using az configure --defaults workspace=<name>.

Optional Parameters

--node-index -i

The index of the node for which the services has to be shown.

Default value: 0
Global Parameters
--debug

Increase logging verbosity to show all debug logs.

--help -h

Show this help message and exit.

--only-show-errors

Only show errors, suppressing warnings.

--output -o

Output format.

Accepted values: json, jsonc, none, table, tsv, yaml, yamlc
Default value: json
--query

JMESPath query string. See http://jmespath.org/ for more information and examples.

--subscription

Name or ID of subscription. You can configure the default subscription using az account set -s NAME_OR_ID.

--verbose

Increase logging verbosity. Use --debug for full debug logs.

az ml job stream

Stream job logs to the console.

az ml job stream --name
                 --resource-group
                 --workspace-name

Required Parameters

--name -n

Name of the job.

--resource-group -g

Name of resource group. You can configure the default group using az configure --defaults group=<name>.

--workspace-name -w

Name of the Azure ML workspace. You can configure the default workspace using az configure --defaults workspace=<name>.

Global Parameters
--debug

Increase logging verbosity to show all debug logs.

--help -h

Show this help message and exit.

--only-show-errors

Only show errors, suppressing warnings.

--output -o

Output format.

Accepted values: json, jsonc, none, table, tsv, yaml, yamlc
Default value: json
--query

JMESPath query string. See http://jmespath.org/ for more information and examples.

--subscription

Name or ID of subscription. You can configure the default subscription using az account set -s NAME_OR_ID.

--verbose

Increase logging verbosity. Use --debug for full debug logs.

az ml job update

Update a job.

Only the 'tags' and 'properties' properties can be updated.

az ml job update --name
                 --resource-group
                 --workspace-name
                 [--add]
                 [--force-string]
                 [--remove]
                 [--set]
                 [--web]

Required Parameters

--name -n

Name of the job.

--resource-group -g

Name of resource group. You can configure the default group using az configure --defaults group=<name>.

--workspace-name -w

Name of the Azure ML workspace. You can configure the default workspace using az configure --defaults workspace=<name>.

Optional Parameters

--add

Add an object to a list of objects by specifying a path and key value pairs. Example: --add property.listProperty <key=value, string or JSON string>.

Default value: []
--force-string

When using 'set' or 'add', preserve string literals instead of attempting to convert to JSON.

Default value: False
--remove

Remove a property or an element from a list. Example: --remove property.list <indexToRemove> OR --remove propertyToRemove.

Default value: []
--set

Update an object by specifying a property path and value to set. Example: --set property1.property2=<value>.

Default value: []
--web -e

Show the job's run details in Azure ML studio in a web browser.

Default value: False
Global Parameters
--debug

Increase logging verbosity to show all debug logs.

--help -h

Show this help message and exit.

--only-show-errors

Only show errors, suppressing warnings.

--output -o

Output format.

Accepted values: json, jsonc, none, table, tsv, yaml, yamlc
Default value: json
--query

JMESPath query string. See http://jmespath.org/ for more information and examples.

--subscription

Name or ID of subscription. You can configure the default subscription using az account set -s NAME_OR_ID.

--verbose

Increase logging verbosity. Use --debug for full debug logs.

az ml job validate

Validate a job. This command works for pipeline jobs only for now.

This command will validate a YAML specification file to check if it is valid for job creation, and return all issues that were found. Validation mainly includes local checking for schema, like missing fields, environment without version specified, code referred to an non-existent local path; it will also check for the existence of referenced compute targets in the target workspace. Validation result will be printed to the console, including both errors and warnings. Only errors will cause the validation to fail. A job passed validation will be able to be submitted. This command works for pipeline jobs only for now.

az ml job validate --file
                   --resource-group
                   --workspace-name
                   [--set]

Examples

Validate a YAML specification file to check if it is valid for job creation.

az ml job validate --file job.yml --resource-group my-resource-group --workspace-name my-workspace

Required Parameters

--file -f

Local path to the YAML file containing the Azure ML job specification. The YAML reference docs for job can be found at: https://aka.ms/ml-cli-v2-job-pipeline-yaml-reference.

--resource-group -g

Name of resource group. You can configure the default group using az configure --defaults group=<name>.

--workspace-name -w

Name of the Azure ML workspace. You can configure the default workspace using az configure --defaults workspace=<name>.

Optional Parameters

--set

Update an object by specifying a property path and value to set. Example: --set property1.property2=.

Global Parameters
--debug

Increase logging verbosity to show all debug logs.

--help -h

Show this help message and exit.

--only-show-errors

Only show errors, suppressing warnings.

--output -o

Output format.

Accepted values: json, jsonc, none, table, tsv, yaml, yamlc
Default value: json
--query

JMESPath query string. See http://jmespath.org/ for more information and examples.

--subscription

Name or ID of subscription. You can configure the default subscription using az account set -s NAME_OR_ID.

--verbose

Increase logging verbosity. Use --debug for full debug logs.