Azure Data Factory Pipelines Rest API How To Create Run with Parameters

Mark P. Hahn 96 Reputation points
2021-08-06T23:42:04.363+00:00

I am trying to use the Azure REST Interface to start an Azure Data Factory Pipeline and supply parameters:
https://learn.microsoft.com/en-us/rest/api/datafactory/pipelines/create-run

The pipeline definition for the pipeline I am referencing has four parameters which are shown when I pull the definition through the Azure REST API:

{
"id": "/subscriptions/blah-blah-blah-blah-blah/resourceGroups/edw-dev/providers/Microsoft.DataFactory/factories/edw-dev-adf/pipelines/PL_TagsProductHeir",
"name": "PL_TagsProductHeir",
"type": "Microsoft.DataFactory/factories/pipelines",
"properties": {
"activities": [
"@{name=DFTagsProductHier; type=ExecuteDataFlow; dependsOn=System.Object[]; policy=; userProperties=System.Object[]; typeProperties=
}"
],
"parameters": {
"PLSourceTable": "@{type=string}",
"PLTargetTable": "@{type=string}",
"PLSourceSchema": "@{type=string}",
"PLTargetSchema": "@{type=string}"
},
"annotations": [],
"lastPublishTime": "2021-08-04T09:10:07Z"
},
"etag": "blah-blah-blah-blah-blah"
}

In PowerShell I am doing the following:

$run_parms=@{PLSourceTable="FFX_TagsProductHierarchy"; PLTargetTable="TagsProductHier"; PLSourceSchema="dbo"; PLTargetSchema="dbo"}
$invoke_reply = Invoke-WebRequest -Method POST -Uri $invoke_uri -Headers $headers -Body $run_parms

However the run never receives these parameters. I know the Uri is correct because the pipeline a pipeline run is created, and also the authentication headers are correct.

So I read this example in the Azure REST API Specification:
https://github.com/Azure/azure-rest-api-specs/blob/main/specification/datafactory/resource-manager/Microsoft.DataFactory/stable/2018-06-01/examples/Pipelines_CreateRun.json
Which led me to try this:

$run_parms=@{PLSourceTable="FFX_TagsProductHierarchy"; PLTargetTable="TagsProductHier"; PLSourceSchema="dbo"; PLTargetSchema="dbo"}
$invoke_params = @{ parameters=$run_parms }
$invoke_reply = Invoke-WebRequest -Method POST -Uri $invoke_uri -Headers $headers -Body $invoke_params

And also this:

$run_parms=@{PLSourceTable="FFX_TagsProductHierarchy"; PLTargetTable="TagsProductHier"; PLSourceSchema="dbo"; PLTargetSchema="dbo"}
$invoke_params = @{ parameters=@{ parameters=$run_parms } }
$invoke_reply = Invoke-WebRequest -Method POST -Uri $invoke_uri -Headers $headers -Body $invoke_params

Based on the REST definition of the pipeline, I tried this:

$run_parms=@{PLSourceTable="FFX_TagsProductHierarchy"; PLTargetTable="TagsProductHier"; PLSourceSchema="dbo"; PLTargetSchema="dbo"}
$invoke_params = @{ properties=@{ parameters=$run_parms } }
$invoke_reply = Invoke-WebRequest -Method POST -Uri $invoke_uri -Headers $headers -Body $invoke_params

Then I somewhat randomly tried this:

$run_parms=@{PLSourceTable="FFX_TagsProductHierarchy"; PLTargetTable="TagsProductHier"; PLSourceSchema="dbo"; PLTargetSchema="dbo"}
$invoke_params = @{ properties=$run_parms }
$invoke_reply = Invoke-WebRequest -Method POST -Uri $invoke_uri -Headers $headers -Body $invoke_params

None of them worked. The pipeline run is created but the parameters never show up in the run.

I have checked the runs by pulling the REST entity for the runs which look like:

{
"id": "/SUBSCRIPTIONS/blah-blah-blah-blah-blah/RESOURCEGROUPS/EDW-DEV/PROVIDERS/MICROSOFT.DATAFACTORY/FACTORIES/EDW-DEV-ADF/pipelineruns/blah-blah-blah-blah-blah",
"runId": "blah-blah-blah-blah-blah",
"debugRunId": null,
"runGroupId": "blah-blah-blah-blah-blah",
"pipelineName": "PL_TagsProductHeir",
"parameters": {
"PLSourceTable": "",
"PLTargetTable": "",
"PLSourceSchema": "",
"PLTargetSchema": ""
},
"invokedBy": { stuff },
"runStart": "2021-08-04T22:15:15.0910311Z",
"runEnd": "2021-08-04T22:20:33.2701997Z",
"durationInMs": 318179,
"status": "Failed",
"message": "",
"output": null,
"lastUpdated": "2021-08-04T22:20:33.2701997Z",
"annotations": [],
"runDimension": {},
"isLatest": true
}

However, if my infrastructure allowed it, I could install the Azure PowerShell module. If that were possible, (its not on the server in question) then this call would work:

$run_parms=@{PLSourceTable="FFX_TagsProductHierarchy"; PLTargetTable="TagsProductHier"; PLSourceSchema="dbo"; PLTargetSchema="dbo"}
Invoke-AzDataFactoryV2Pipeline -ResourceGroupName edw-dev -DataFactoryName edw-dev-adf -PipelineName PL_TagsProductHeir -Parameter $run_parms

Please don't make me read the source code for the Invoke-AzDataFactoryV2Pipeline cmdlet to figure out how to format the pipeline parameters.

The successful runs have parameters when I pull their run information via REST:

. . . .
"parameters": {
"PLSourceTable": "FFX_TagsProductHierarchy",
"PLTargetTable": "TagsProductHier",
"PLSourceSchema": "dbo",
"PLTargetSchema": "dob"
},
. . . .
"status": "Succeeded",
. . . .

What am I doing wrong and how am I misreading the documentation?

Thank you, -MpH

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,811 questions
{count} votes

Accepted answer
  1. Mark P. Hahn 96 Reputation points
    2021-08-12T00:02:55.157+00:00

    Himanshu,

    I looked at your example code and it is wrong. In your urivalue you have . . . /pipelines/parameter/ . . ..
    You are including /parameter/ ( I assume you meant /parameter/) in the uri path. There is no such entity in the swagger specification:
    https://github.com/Azure/azure-rest-api-specs/blob/88e7838a09868a51de3894114355c75929847a46/specification/datafactory/resource-manager/Microsoft.DataFactory/stable/2018-06-01/datafactory.json#L2382

    However, I did see my mistake. I am passing a PowerShell hashtable object as the body of the web request:

    $run_parms=@{fred="flintsone3"; barney="rubble"}
    $invoke_reply = Invoke-WebRequest -Method Post -Uri $invoke_uri -Headers $headers -Body $run_parms
    

    I assumed that the Invoke-WebRequest cmdlet would convert the
    $run_parms to json. That was wrong. Even Invoke-RestMethod does
    not do this conversion.

    The argument $run_parms must be converted to json mannually like so:

    $run_parms=@{fred="flintsone3"; barney="rubble"}
    $json_run_parms = ConvertTo-Json -InputObject $run_parms
    $invoke_reply = Invoke-WebRequest -Method Post -Uri $invoke_uri -Headers $headers -Body $json_run_parms
    

    This will work as expected.

    Thank you for your assistance.

    0 comments No comments

3 additional answers

Sort by: Most helpful
  1. HimanshuSinha-msft 19,476 Reputation points Microsoft Employee
    2021-08-09T21:16:21.703+00:00

    Hello @Mark P. Hahn ,
    Thanks for the ask and using the Microsoft Q&A platform .
    I did tried out the ask , basically you are having a tough time in passing the parameters to the pipeline , ( atleast thats what i understood ) .
    I created a basic pipeline with two parameters which just sets the varaiables with the incoming parameter .

    121750-image.png

    $urivalue ='https://management.azure.com/subscriptions/XXXXX/resourceGroups/Analytics-Resources/providers/Microsoft.DataFactory/factories/Myfactory/pipelines/paramter/createRun?api-version=2018-06-01'
    $header =@{'Authorization'='Bearer XXXX'; 'Content-type'= 'application/json' }
    $body = '{"myparamaname1":"someparam1","myparamaname2":"someparam3"}'
    Invoke-WebRequest -Uri $urivalue -Headers $header -Method Post -Body $body

    The output which i recieved is

    tatusCode : 200
    StatusDescription : OK
    Content : {"runId":"7e72b0b7-f291-451b-ad46-b7eec70e6284"}
    RawContent : HTTP/1.1 200 OK
    Pragma: no-cache
    Strict-Transport-Security: max-age=31536000; includeSubDomains
    X-Content-Type-Options: nosniff
    x-ms-ratelimit-remaining-subscription-writes: 1196
    x-ms-request-id:...
    Forms : {}
    Headers : {[Pragma, no-cache], [Strict-Transport-Security, max-age=31536000;
    includeSubDomains], [X-Content-Type-Options, nosniff],
    [x-ms-ratelimit-remaining-subscription-writes, 1196]...}
    Images : {}
    InputFields : {}
    Links : {}
    ParsedHtml : mshtml.HTMLDocumentClass
    RawContentLength : 48

    I see that the pipeline did captured the parameters .
    121801-image.png

    Please do let me know how it goes .
    Thanks
    Himanshu
    Please do consider clicking on "Accept Answer" and "Up-vote" on the post that helps you, as it can be beneficial to other community members

    1 person found this answer helpful.

  2. Mark P. Hahn 96 Reputation points
    2021-08-11T13:40:42.527+00:00

    I have not yet had a chance to try this. I will shortly. Thanks for the clear instructions. -MpH

    0 comments No comments

  3. S Quinn 1 Reputation point
    2022-04-01T20:40:52.263+00:00

    I got it to work like this:

    $params=@{"parameter1Name"="parameter1value"
    }| ConvertTo-Json

    $headers = @{
    'Authorization' = "Bearer $token"
    'content_type' = "application/json"
    }

    $responseADF = Invoke-RestMethod -Uri $uri -Method Post -Headers $headers -Body $params -ContentType "application/json"
    write-host $responseADF

    0 comments No comments

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.