Execute pipeline from pyspark

arkiboys 9,556 Reputation points
2022-02-02T20:22:44.013+00:00

Hi,
In serverless sql pool, I have a pyspark notebook in which I would like to execute the pipeline, pipeline1.
How is this possible?

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
4,348 questions
{count} votes

Accepted answer
  1. HimanshuSinha-msft 19,376 Reputation points Microsoft Employee
    2022-02-04T00:55:20.247+00:00

    Hello @arkiboys ,
    Thanks for the ask and using Microsoft Q&A platform .
    As we understand the ask here is to invoke a existing pipeline from pyspark . Please do let me know if that not accurate.
    You can use the REST API call to the invoke the pipeline from the pyspark . The API which you will need to invoke is Pipeline - Create Pipeline Run

    You will need a

    1. Service prinicipal ( client ID and client secret ) and this service prinicipal should have acess on the workspace .
    2. Tenantid
    3. Workspace Name

    It has two steps

    1. Get the bearer token
    2. Execute the API

    https://learn.microsoft.com/en-us/rest/api/synapse/data-plane/pipeline/create-pipeline-run

    The below code will return the runid of the pipeline which you are trying to invoke .

    import http.client
    import json
    conn = http.client.HTTPSConnection("login.microsoftonline.com")
    payload = 'grant_type=client_credentials&client_id={Yourclientid}&client_secret={client secret}&resource=https%3A%2F%2Fdev.azuresynapse.net%2F'
    conn.request("POST"
    , "/{tenantID}/oauth2/token"
    , payload
    , headers)
    res = conn.getresponse()
    data = res.read()
    Response = json.loads(data)
    AccessToken = Response["access_token"]

    conn = http.client.HTTPSConnection("{YourWorkspace}.dev.azuresynapse.net")
    payload = ''
    headers = {
    'Authorization': ' Bearer '+ AccessToken
    }
    conn.request("POST", "/pipelines/Eventtrigger/createRun?api-version=2020-12-01", payload, headers)
    res = conn.getresponse()
    data = res.read()
    print(data.decode("utf-8"))

    171139-image.png

    171203-image.png

    Please do let me if you have any queries .
    Thanks
    Himanshu

    -------------------------------------------------------------------------------------------------------------------------

    • Please don't forget to click on 130616-image.png or upvote 130671-image.png button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
    • Want a reminder to come back and check responses? Here is how to subscribe to a notification
    • If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators
    0 comments No comments

0 additional answers

Sort by: Most helpful