I am trying to use the Azure REST Interface to start an Azure Data Factory Pipeline and supply parameters:
https://learn.microsoft.com/en-us/rest/api/datafactory/pipelines/create-run
The pipeline definition for the pipeline I am referencing has four parameters which are shown when I pull the definition through the Azure REST API:
{
"id": "/subscriptions/blah-blah-blah-blah-blah/resourceGroups/edw-dev/providers/Microsoft.DataFactory/factories/edw-dev-adf/pipelines/PL_TagsProductHeir",
"name": "PL_TagsProductHeir",
"type": "Microsoft.DataFactory/factories/pipelines",
"properties": {
"activities": [
"@{name=DFTagsProductHier; type=ExecuteDataFlow; dependsOn=System.Object[]; policy=; userProperties=System.Object[]; typeProperties=
}"
],
"parameters": {
"PLSourceTable": "@{type=string}",
"PLTargetTable": "@{type=string}",
"PLSourceSchema": "@{type=string}",
"PLTargetSchema": "@{type=string}"
},
"annotations": [],
"lastPublishTime": "2021-08-04T09:10:07Z"
},
"etag": "blah-blah-blah-blah-blah"
}
In PowerShell I am doing the following:
$run_parms=@{PLSourceTable="FFX_TagsProductHierarchy"; PLTargetTable="TagsProductHier"; PLSourceSchema="dbo"; PLTargetSchema="dbo"}
$invoke_reply = Invoke-WebRequest -Method POST -Uri $invoke_uri -Headers $headers -Body $run_parms
However the run never receives these parameters. I know the Uri is correct because the pipeline a pipeline run is created, and also the authentication headers are correct.
So I read this example in the Azure REST API Specification:
https://github.com/Azure/azure-rest-api-specs/blob/main/specification/datafactory/resource-manager/Microsoft.DataFactory/stable/2018-06-01/examples/Pipelines_CreateRun.json
Which led me to try this:
$run_parms=@{PLSourceTable="FFX_TagsProductHierarchy"; PLTargetTable="TagsProductHier"; PLSourceSchema="dbo"; PLTargetSchema="dbo"}
$invoke_params = @{ parameters=$run_parms }
$invoke_reply = Invoke-WebRequest -Method POST -Uri $invoke_uri -Headers $headers -Body $invoke_params
And also this:
$run_parms=@{PLSourceTable="FFX_TagsProductHierarchy"; PLTargetTable="TagsProductHier"; PLSourceSchema="dbo"; PLTargetSchema="dbo"}
$invoke_params = @{ parameters=@{ parameters=$run_parms } }
$invoke_reply = Invoke-WebRequest -Method POST -Uri $invoke_uri -Headers $headers -Body $invoke_params
Based on the REST definition of the pipeline, I tried this:
$run_parms=@{PLSourceTable="FFX_TagsProductHierarchy"; PLTargetTable="TagsProductHier"; PLSourceSchema="dbo"; PLTargetSchema="dbo"}
$invoke_params = @{ properties=@{ parameters=$run_parms } }
$invoke_reply = Invoke-WebRequest -Method POST -Uri $invoke_uri -Headers $headers -Body $invoke_params
Then I somewhat randomly tried this:
$run_parms=@{PLSourceTable="FFX_TagsProductHierarchy"; PLTargetTable="TagsProductHier"; PLSourceSchema="dbo"; PLTargetSchema="dbo"}
$invoke_params = @{ properties=$run_parms }
$invoke_reply = Invoke-WebRequest -Method POST -Uri $invoke_uri -Headers $headers -Body $invoke_params
None of them worked. The pipeline run is created but the parameters never show up in the run.
I have checked the runs by pulling the REST entity for the runs which look like:
{
"id": "/SUBSCRIPTIONS/blah-blah-blah-blah-blah/RESOURCEGROUPS/EDW-DEV/PROVIDERS/MICROSOFT.DATAFACTORY/FACTORIES/EDW-DEV-ADF/pipelineruns/blah-blah-blah-blah-blah",
"runId": "blah-blah-blah-blah-blah",
"debugRunId": null,
"runGroupId": "blah-blah-blah-blah-blah",
"pipelineName": "PL_TagsProductHeir",
"parameters": {
"PLSourceTable": "",
"PLTargetTable": "",
"PLSourceSchema": "",
"PLTargetSchema": ""
},
"invokedBy": { stuff },
"runStart": "2021-08-04T22:15:15.0910311Z",
"runEnd": "2021-08-04T22:20:33.2701997Z",
"durationInMs": 318179,
"status": "Failed",
"message": "",
"output": null,
"lastUpdated": "2021-08-04T22:20:33.2701997Z",
"annotations": [],
"runDimension": {},
"isLatest": true
}
However, if my infrastructure allowed it, I could install the Azure PowerShell module. If that were possible, (its not on the server in question) then this call would work:
$run_parms=@{PLSourceTable="FFX_TagsProductHierarchy"; PLTargetTable="TagsProductHier"; PLSourceSchema="dbo"; PLTargetSchema="dbo"}
Invoke-AzDataFactoryV2Pipeline -ResourceGroupName edw-dev -DataFactoryName edw-dev-adf -PipelineName PL_TagsProductHeir -Parameter $run_parms
Please don't make me read the source code for the Invoke-AzDataFactoryV2Pipeline cmdlet to figure out how to format the pipeline parameters.
The successful runs have parameters when I pull their run information via REST:
. . . .
"parameters": {
"PLSourceTable": "FFX_TagsProductHierarchy",
"PLTargetTable": "TagsProductHier",
"PLSourceSchema": "dbo",
"PLTargetSchema": "dob"
},
. . . .
"status": "Succeeded",
. . . .
What am I doing wrong and how am I misreading the documentation?
Thank you, -MpH