Synapse Pipeline Configs, URL's - How to prep for production?

Giri, Seshu 26 Reputation points
2022-10-21T00:56:15.487+00:00

Hi,

How do I prepare Synapse pipeline's for production? For example,

1. I am using a web activity to read a sceret from Key Vault. How do I change the URL when moved to prod? Notice in the URL - I have UAT https://company-keyvault-uat.vault.azure.net/secrets/SomeID?api-version=7.0

252751-image.png

2. How about Linked Services which talk to storage accounts, SQL DB etc?

Azure Key Vault
Azure Key Vault
An Azure service that is used to manage and protect cryptographic keys and other secrets used by cloud apps and services.
1,282 questions
Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
4,912 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,663 questions
{count} votes

4 answers

Sort by: Most helpful
  1. Harish Sune 6 Reputation points
    2022-10-26T15:42:52.49+00:00

    Sure @Giri, Seshu , Detailed article here - https://iterationinsights.com/article/azure-data-factory-ci-cd-with-devops-pipelines/ you can refer.

    Please feel free to reach out if you need any explanation related same.

    • Please don't forget to click on 130616-image.png or upvote 130671-image.png button whenever the information provided helps you.
    1 person found this answer helpful.
    0 comments No comments

  2. MartinJaffer-MSFT 26,086 Reputation points
    2022-10-25T04:57:23.89+00:00

    Hello @Giri, Seshu ,
    Thanks for the question and using MS Q&A platform.

    I understand you want to parameterize the URL in a web activity for different environments.

    I think the easiest solution, is to use Global Parameters, in this case.

    A global parameter is made available to any and all pipelines in a Factory. When you change the value of a global parameter, it then affects ALL places it is used.
    You can make the global parameters one of the things you set when doing CI/CD or moving dev -> test -> prod.

    There are a couple ways you could use it. The most straightforward is to store the appropriate URL , and then reference it in the web activity like @concat(pipeline().globalParameter.KeyVaultURL , "secrets/SomeID?api-version=7.0) .

    A more convoluted way, is to store a lablel indicating which environment this is. Then in the web activity URL you could put an expression like

    @if(  
      eq( "prod" , pipeline().globalParameter.environment ),  
      "MyProdKeyvaultURL",  
      "MyDevKevaultURL")  
    

    I personally recommend the simpler method.

    There is a way to set the URL itself as something to parameterize in the ARM template, but this would effect ALL web activities, not just this one. Thus I recommend the global parameter route.

    Please do let me if you have any queries.

    Thanks
    Martin


    • Please don't forget to click on 130616-image.png or upvote 130671-image.png button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
    • Want a reminder to come back and check responses? Here is how to subscribe to a notification
      • If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators
    0 comments No comments

  3. Giri, Seshu 26 Reputation points
    2022-10-26T14:10:11.233+00:00

    Hi @MartinJaffer-MSFT ,

    I am using Synapse, there is no support for Global parameters. Any other idea?


  4. Harish Sune 6 Reputation points
    2022-10-26T14:52:45.843+00:00

    Hi @Giri, Seshu you can achieve this by using release pipelines of Azure. You can add stages and pipeline variables based on the environment to achieve this.

    254369-image.png

    Let me know if you need more details about creating release pipeline, also please accept the answer if this help.

    • Please don't forget to click on 130616-image.png or upvote 130671-image.png button whenever the information provided helps you.

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.