Executing pipelines dynamically

Question

Hi,
I have a view in serverless sql pool, view1.
The view1 contains list of pipeline names.
How is it possible to execute the pipelines in this table?
Thankyou

Answer 1

Hi,

How is it possible to execute the pipelines in this table?

If I understand your description correctly, then your table does not include any pipeline (no pipeline JSON definition) but only the pipeline names. Meaning you have pipelines and in your table/view you store their names and you want to execute these pipelines by their name.

You can manually run your pipeline using:
* .NET SDK
* Azure PowerShell module
* REST API
* Python SDK

The specific solution for your request depend on the way you want to execute the pipeline but the simple procedure will be

(1) connect the database and get the names of the pipelines you want to execute

(2) Loop on all the names and for each one execute the pipeline by it's name

For this step you can get sample code here:

https://learn.microsoft.com/en-us/azure/data-factory/concepts-pipeline-execution-triggers

For example if you want to use PowerShell then you need to execute the command Invoke-AzDataFactoryV2Pipeline. For example:

Invoke-AzDataFactoryV2Pipeline -ResourceGroupName "ADF" -DataFactoryName "WikiADF" -PipelineName "MyPipeLine"  

In the above command you can change the Resource Group name, Data Factory Name and the name of the pipeline MyPipeLine to the one you got from the table

https://learn.microsoft.com/en-us/powershell/module/az.datafactory/invoke-azdatafactoryv2pipeline

Answer 2

Hello @arkiboys ,
You can use a lookup to get the output of view and then pass them to for each loop . In the for each loop you can use two web activity , one for getting the bearer token and one for running the API ( which will use the bear token .
I have done the same thing use notebook below .
https://learn.microsoft.com/en-us/answers/questions/720372/execute-pipeline-from-pyspark.html?childToView=722295#answer-722295
HTH
Thanks
Himanshu