Follow these key steps to integrate ADF with Azure DevOps and establish an approval-based deployment pipeline:
1. Connect Azure Data Factory to Azure DevOps for Source Control
ADF supports Git repositories, including Azure DevOps Git and GitHub, for source control. Here's how to link your ADF to an Azure DevOps Git repository:
- Open Azure Data Factory Studio: In the ADF portal, go to the Author & Monitor section.
- Set Up Git Integration:
- In the top menu, click on the Manage hub (the gear icon).
- Under Git Configuration, select Azure DevOps Git as the repository type.
- Enter details such as Azure DevOps organization, project, repository name, collaboration branch (usually
main
ormaster
), and root folder.
- Publish from Collaboration Branch: Once integrated, any changes made in the ADF studio are automatically committed to the collaboration branch in Azure DevOps. You can publish changes from the Collaboration branch to the Data Factory’s service instance by clicking Publish All.
- Branching: You can use feature branches for development and then merge changes into the collaboration branch via pull requests.
You can follow this Microsoft guide for version control setup: Set up source control in Azure Data Factory
2. Set Up Azure Pipelines for CI/CD
To move code from development to other environments (such as QA, production), set up a Continuous Integration (CI) and Continuous Delivery (CD) pipeline in Azure DevOps.
- Create a Pipeline:
- In Azure DevOps, navigate to Pipelines.
- Click New Pipeline and choose the source repository where ADF code is stored.
- Define a pipeline using the YAML pipeline editor or use the Classic Editor for a GUI-based approach.
- You can use Azure Resource Manager (ARM) templates to deploy Data Factory objects (datasets, pipelines, etc.) across environments.
- Pipeline Configuration:
- The ADF service automatically generates ARM templates when you publish changes. These templates are stored in the repository's
adf_publish
branch. - Use these templates in your deployment pipeline to deploy changes to higher environments (QA/Prod).
- Example of YAML pipeline:
trigger: branches: include: - master pool: vmImage: 'windows-latest' steps: - task: AzureResourceManagerTemplateDeployment@3 inputs: deploymentScope: 'Resource Group' azureResourceManagerConnection: 'AzureRMServiceConnection' action: 'Create Or Update Resource Group' resourceGroupName: 'YourResourceGroup' location: 'YourLocation' templateLocation: 'Linked artifact' csmFile: '$(System.DefaultWorkingDirectory)/adf_publish/ARMTemplateForFactory.json' csmParametersFile: '$(System.DefaultWorkingDirectory)/adf_publish/ARMTemplateParametersForFactory.json'
- The ADF service automatically generates ARM templates when you publish changes. These templates are stored in the repository's
3. Implement an Approval Process
Azure DevOps allows you to add manual intervention or approval steps before deploying to specific environments.
- Set up Environments with Approvals:
- Go to the Pipelines > Environments section in Azure DevOps.
- Create a new environment (e.g., QA, Production).
- Configure approvers for that environment, who will be notified to approve deployments before they proceed.
- In your pipeline, use a stage for each environment and specify the environment configured for approvals.
stages:
- stage: QA
jobs:
- deployment: QA_Deploy
environment: 'QA'
strategy:
runOnce:
deploy:
steps:
- script: echo Deploying to QA
More links :
Configure Git integration with Azure Data Factory
Continuous integration and delivery in Azure Data Factory
Azure Pipelines Environments and Approvals
These steps will enable you to set up version control, configure CI/CD pipelines, and implement an approval process for deploying ADF assets across different environments.