az datafactory pipeline

  • Reference

Note

This reference is part of the datafactory extension for the Azure CLI (version 2.15.0 or higher). The extension will automatically install the first time you run an az datafactory pipeline command. Learn more about extensions.

Manage pipeline with datafactory.

Commands

Name Description Type Status
az datafactory pipeline create

Create a pipeline.

 Extension GA
az datafactory pipeline create-run

Creates a run of a pipeline.

 Extension GA
az datafactory pipeline delete

Deletes a pipeline.

 Extension GA
az datafactory pipeline list

Lists pipelines.

 Extension GA
az datafactory pipeline show

Gets a pipeline.

 Extension GA
az datafactory pipeline update

Update a pipeline.

 Extension GA

az datafactory pipeline create

Create a pipeline.

az datafactory pipeline create --factory-name
                               --name
                               --pipeline
                               --resource-group
                               [--if-match]

Examples

Pipelines_Create

az datafactory pipeline create --factory-name "exampleFactoryName" --pipeline "{\"activities\":[{\"name\":\"ExampleForeachActivity\",\"type\":\"ForEach\",\"typeProperties\":{\"activities\":[{\"name\":\"ExampleCopyActivity\",\"type\":\"Copy\",\"inputs\":[{\"type\":\"DatasetReference\",\"parameters\":{\"MyFileName\":\"examplecontainer.csv\",\"MyFolderPath\":\"examplecontainer\"},\"referenceName\":\"exampleDataset\"}],\"outputs\":[{\"type\":\"DatasetReference\",\"parameters\":{\"MyFileName\":{\"type\":\"Expression\",\"value\":\"@item()\"},\"MyFolderPath\":\"examplecontainer\"},\"referenceName\":\"exampleDataset\"}],\"typeProperties\":{\"dataIntegrationUnits\":32,\"sink\":{\"type\":\"BlobSink\"},\"source\":{\"type\":\"BlobSource\"}}}],\"isSequential\":true,\"items\":{\"type\":\"Expression\",\"value\":\"@pipeline().parameters.OutputBlobNameList\"}}}],\"parameters\":{\"JobId\":{\"type\":\"String\"},\"OutputBlobNameList\":{\"type\":\"Array\"}},\"variables\":{\"TestVariableArray\":{\"type\":\"Array\"}},\"runDimensions\":{\"JobId\":{\"type\":\"Expression\",\"value\":\"@pipeline().parameters.JobId\"}},\"duration\":\"0.00:10:00\"}" --name "examplePipeline" --resource-group "exampleResourceGroup"

Required Parameters

--factory-name

The factory name.

--name --pipeline-name -n

The pipeline name.

--pipeline

Pipeline resource definition. Expected value: json-string/json-file/@json-file.

--resource-group -g

Name of resource group. You can configure the default group using az configure --defaults group=<name>.

Optional Parameters

--if-match

ETag of the pipeline entity. Should only be specified for update, for which it should match existing entity or can be * for unconditional update.

Global Parameters
--debug

Increase logging verbosity to show all debug logs.

--help -h

Show this help message and exit.

--only-show-errors

Only show errors, suppressing warnings.

--output -o

Output format.

accepted values: json, jsonc, none, table, tsv, yaml, yamlc
default value: json
--query

JMESPath query string. See http://jmespath.org/ for more information and examples.

--subscription

Name or ID of subscription. You can configure the default subscription using az account set -s NAME_OR_ID.

--verbose

Increase logging verbosity. Use --debug for full debug logs.

az datafactory pipeline create-run

Creates a run of a pipeline.

az datafactory pipeline create-run --factory-name
                                   --name
                                   --resource-group
                                   [--is-recovery {false, true}]
                                   [--parameters]
                                   [--reference-pipeline-run-id]
                                   [--start-activity-name]
                                   [--start-from-failure {false, true}]

Examples

Pipelines_CreateRun

az datafactory pipeline create-run --factory-name "exampleFactoryName" --parameters "{\"OutputBlobNameList\":[\"exampleoutput.csv\"]}" --name "examplePipeline" --resource-group "exampleResourceGroup"

Required Parameters

--factory-name

The factory name.

--name --pipeline-name -n

The pipeline name.

--resource-group -g

Name of resource group. You can configure the default group using az configure --defaults group=<name>.

Optional Parameters

--is-recovery

Recovery mode flag. If recovery mode is set to true, the specified referenced pipeline run and the new run will be grouped under the same groupId.

accepted values: false, true
--parameters

Parameters of the pipeline run. These parameters will be used only if the runId is not specified. Expected value: json-string/json-file/@json-file.

--reference-pipeline-run-id

The pipeline run identifier. If run ID is specified the parameters of the specified run will be used to create a new run.

--start-activity-name

In recovery mode, the rerun will start from this activity. If not specified, all activities will run.

--start-from-failure

In recovery mode, if set to true, the rerun will start from failed activities. The property will be used only if startActivityName is not specified.

accepted values: false, true
Global Parameters
--debug

Increase logging verbosity to show all debug logs.

--help -h

Show this help message and exit.

--only-show-errors

Only show errors, suppressing warnings.

--output -o

Output format.

accepted values: json, jsonc, none, table, tsv, yaml, yamlc
default value: json
--query

JMESPath query string. See http://jmespath.org/ for more information and examples.

--subscription

Name or ID of subscription. You can configure the default subscription using az account set -s NAME_OR_ID.

--verbose

Increase logging verbosity. Use --debug for full debug logs.

az datafactory pipeline delete

Deletes a pipeline.

az datafactory pipeline delete [--factory-name]
                               [--ids]
                               [--name]
                               [--resource-group]
                               [--subscription]
                               [--yes]

Examples

Pipelines_Delete

az datafactory pipeline delete --factory-name "exampleFactoryName" --name "examplePipeline" --resource-group "exampleResourceGroup"

Optional Parameters

--factory-name

The factory name.

--ids

One or more resource IDs (space-delimited). It should be a complete resource ID containing all information of 'Resource Id' arguments. You should provide either --ids or other 'Resource Id' arguments.

--name --pipeline-name -n

The pipeline name.

--resource-group -g

Name of resource group. You can configure the default group using az configure --defaults group=<name>.

--subscription

Name or ID of subscription. You can configure the default subscription using az account set -s NAME_OR_ID.

--yes -y

Do not prompt for confirmation.

default value: False
Global Parameters
--debug

Increase logging verbosity to show all debug logs.

--help -h

Show this help message and exit.

--only-show-errors

Only show errors, suppressing warnings.

--output -o

Output format.

accepted values: json, jsonc, none, table, tsv, yaml, yamlc
default value: json
--query

JMESPath query string. See http://jmespath.org/ for more information and examples.

--subscription

Name or ID of subscription. You can configure the default subscription using az account set -s NAME_OR_ID.

--verbose

Increase logging verbosity. Use --debug for full debug logs.

az datafactory pipeline list

Lists pipelines.

az datafactory pipeline list --factory-name
                             --resource-group

Examples

Pipelines_ListByFactory

az datafactory pipeline list --factory-name "exampleFactoryName" --resource-group "exampleResourceGroup"

Required Parameters

--factory-name

The factory name.

--resource-group -g

Name of resource group. You can configure the default group using az configure --defaults group=<name>.

Global Parameters
--debug

Increase logging verbosity to show all debug logs.

--help -h

Show this help message and exit.

--only-show-errors

Only show errors, suppressing warnings.

--output -o

Output format.

accepted values: json, jsonc, none, table, tsv, yaml, yamlc
default value: json
--query

JMESPath query string. See http://jmespath.org/ for more information and examples.

--subscription

Name or ID of subscription. You can configure the default subscription using az account set -s NAME_OR_ID.

--verbose

Increase logging verbosity. Use --debug for full debug logs.

az datafactory pipeline show

Gets a pipeline.

az datafactory pipeline show [--factory-name]
                             [--ids]
                             [--if-none-match]
                             [--name]
                             [--resource-group]
                             [--subscription]

Examples

Pipelines_Get

az datafactory pipeline show --factory-name "exampleFactoryName" --name "examplePipeline" --resource-group "exampleResourceGroup"

Optional Parameters

--factory-name

The factory name.

--ids

One or more resource IDs (space-delimited). It should be a complete resource ID containing all information of 'Resource Id' arguments. You should provide either --ids or other 'Resource Id' arguments.

--if-none-match

ETag of the pipeline entity. Should only be specified for get. If the ETag matches the existing entity tag, or if * was provided, then no content will be returned.

--name --pipeline-name -n

The pipeline name.

--resource-group -g

Name of resource group. You can configure the default group using az configure --defaults group=<name>.

--subscription

Name or ID of subscription. You can configure the default subscription using az account set -s NAME_OR_ID.

Global Parameters
--debug

Increase logging verbosity to show all debug logs.

--help -h

Show this help message and exit.

--only-show-errors

Only show errors, suppressing warnings.

--output -o

Output format.

accepted values: json, jsonc, none, table, tsv, yaml, yamlc
default value: json
--query

JMESPath query string. See http://jmespath.org/ for more information and examples.

--subscription

Name or ID of subscription. You can configure the default subscription using az account set -s NAME_OR_ID.

--verbose

Increase logging verbosity. Use --debug for full debug logs.

az datafactory pipeline update

Update a pipeline.

az datafactory pipeline update [--activities]
                               [--add]
                               [--annotations]
                               [--concurrency]
                               [--description]
                               [--duration]
                               [--factory-name]
                               [--folder-name]
                               [--force-string]
                               [--ids]
                               [--if-match]
                               [--if-none-match]
                               [--name]
                               [--parameters]
                               [--remove]
                               [--resource-group]
                               [--run-dimensions]
                               [--set]
                               [--subscription]
                               [--variables]

Examples

Pipelines_Update

az datafactory pipeline update --factory-name "exampleFactoryName" --description "Example description" --activities "[{\"name\":\"ExampleForeachActivity\",\"type\":\"ForEach\",\"typeProperties\":{\"activities\":[{\"name\":\"ExampleCopyActivity\",\"type\":\"Copy\",\"inputs\":[{\"type\":\"DatasetReference\",\"parameters\":{\"MyFileName\":\"examplecontainer.csv\",\"MyFolderPath\":\"examplecontainer\"},\"referenceName\":\"exampleDataset\"}],\"outputs\":[{\"type\":\"DatasetReference\",\"parameters\":{\"MyFileName\":{\"type\":\"Expression\",\"value\":\"@item()\"},\"MyFolderPath\":\"examplecontainer\"},\"referenceName\":\"exampleDataset\"}],\"typeProperties\":{\"dataIntegrationUnits\":32,\"sink\":{\"type\":\"BlobSink\"},\"source\":{\"type\":\"BlobSource\"}}}],\"isSequential\":true,\"items\":{\"type\":\"Expression\",\"value\":\"@pipeline().parameters.OutputBlobNameList\"}}}]" --parameters "{\"OutputBlobNameList\":{\"type\":\"Array\"}}" --duration "0.00:10:00" --name "examplePipeline" --resource-group "exampleResourceGroup"

Optional Parameters

--activities

List of activities in pipeline. Expected value: json-string/json-file/@json-file.

--add

Add an object to a list of objects by specifying a path and key value pairs. Example: --add property.listProperty <key=value, string or JSON string>.

default value: []
--annotations

List of tags that can be used for describing the Pipeline. Expected value: json-string/json-file/@json-file.

--concurrency

The max number of concurrent runs for the pipeline.

--description

The description of the pipeline.

--duration

TimeSpan value, after which an Azure Monitoring Metric is fired. Expected value: json-string/json-file/@json-file.

--factory-name

The factory name.

--folder-name

The name of the folder that this Pipeline is in.

--force-string

When using 'set' or 'add', preserve string literals instead of attempting to convert to JSON.

default value: False
--ids

One or more resource IDs (space-delimited). It should be a complete resource ID containing all information of 'Resource Id' arguments. You should provide either --ids or other 'Resource Id' arguments.

--if-match

ETag of the pipeline entity. Should only be specified for update, for which it should match existing entity or can be * for unconditional update.

--if-none-match

ETag of the pipeline entity. Should only be specified for get. If the ETag matches the existing entity tag, or if * was provided, then no content will be returned. Default value is None.

--name --pipeline-name -n

The pipeline name.

--parameters

List of parameters for pipeline. Expected value: json-string/json-file/@json-file.

--remove

Remove a property or an element from a list. Example: --remove property.list <indexToRemove> OR --remove propertyToRemove.

default value: []
--resource-group -g

Name of resource group. You can configure the default group using az configure --defaults group=<name>.

--run-dimensions

Dimensions emitted by Pipeline. Expected value: json-string/json-file/@json-file.

--set

Update an object by specifying a property path and value to set. Example: --set property1.property2=<value>.

default value: []
--subscription

Name or ID of subscription. You can configure the default subscription using az account set -s NAME_OR_ID.

--variables

List of variables for pipeline. Expected value: json-string/json-file/@json-file.

Global Parameters
--debug

Increase logging verbosity to show all debug logs.

--help -h

Show this help message and exit.

--only-show-errors

Only show errors, suppressing warnings.

--output -o

Output format.

accepted values: json, jsonc, none, table, tsv, yaml, yamlc
default value: json
--query

JMESPath query string. See http://jmespath.org/ for more information and examples.

--subscription

Name or ID of subscription. You can configure the default subscription using az account set -s NAME_OR_ID.

--verbose

Increase logging verbosity. Use --debug for full debug logs.