Call Synapse pipeline with a notebook activity

APPLIES TO: Azure Data Factory Azure Synapse Analytics

Tip

Try out Data Factory in Microsoft Fabric, an all-in-one analytics solution for enterprises. Microsoft Fabric covers everything from data movement to data science, real-time analytics, business intelligence, and reporting. Learn how to start a new trial for free!

In this tutorial, you create an end-to-end pipeline that contains the Web, Until, and Fail activities in Azure Data Factory.

• Web calls a Synapse pipeline with a notebook activity.

• Until gets Synapse pipeline status until completion (status output as Succeeded, Failed, or canceled).

• Fail fails activity and customizes both error message and error code.

Prerequisites

• You have a Synapse pipeline with a notebook activity.
• Use role-based access control(RBAC) to grant permissions for Data Factory to call Synapse workspace endpoint, details refer to Managed Identity.
• Grant permissions for Data Factory to execute/run Synapse pipeline, details refer to Synapse RBAC roles.

How to use this template

1. Go to the Call Synapse pipeline with a notebook activity template.

2. Select Use this template. You'll see a pipeline created.

   

Pipeline introduction and configuration

Review the configurations of your pipeline and make any necessary changes.

1. Pipeline parameters. Change settings if necessary.

   • SynapseEndpoint: The workspace development endpoint, for example https://myworkspace.dev.azuresynapse.net.

   • PLName: Synapse pipeline name in Synapse workspace to execute.

   • FailedOrCancelledErrorCode: Customize error code if Synapse pipeline fails.

     

2. A Web activity to call Synapse pipeline.

   • URL: @Concat(pipeline().parameters.SynapseEndpoint,'/pipelines/',pipeline().parameters.PLName,'/createRun?api-version=2020-12-01'). Refer to Create pipeline run via Synapse REST API.

   • Authentication: Managed Identity is used for authentication. Refer to Prerequisites to grant necessary permissions.

   • Resource : https://dev.azuresynapse.net/.

     

3. Do Web activity to get Synapse pipeline status, Until web activity returns status of Succeeded, Failed, or Canceled.

   • Web URL: @concat(pipeline().parameters.SynapseEndpoint,'/pipelineruns/',activity('Call Synapse Pipeline with Notebook').output.runId,'?api-version=2020-12-01'). Refer to Synapse Get Pipeline Run REST API.

     

   • Until expression: @or(or(equals('Succeeded',activity('Get run status').output.status),equals('Failed',activity('Get run status').output.status)),equals('Cancelled',activity('Get run status').output.status)).

     

4. An If Condition activity evaluates pipeline run status.

   • Expression": @equals('Failed',activity('Get run status').output.status)

   

5. A Fail activity to fail the pipeline.

   • Fail message: @activity('Get run status').output.message

   • Error code: @pipeline().parameters.FailedOrCancelledErrorCode

     

Related content

• Overview of templates