Integrate ADF Pipelines with Azure DevOps

Question

Hi All,

I have built close to 8 pipelines in ADF. Now, I need to integrate all my pipelines with Azure DevOps and going forward want to build the pipelines as features through Azure DevOps.

Kindly, let me know as how I can integrate all my old ADF pipelines with Azure DevOps.

Regards,

Mohanraj

Answer 1

With Azure DevOps will to adopt a CI/CD approach for your data pipelines.

First start with setting up the code repository in ADF, go to your ADF instance in the Azure Portal and choose the Azure DevOps Git (or GitHub, if you prefer) and fill in the necessary details for the repository, branch...

All your existing pipelines and datasets will be in the Azure DevOps repository.

Then navigate to your Azure DevOps account where you will create a new project where you can manage the source code of your ADF pipelines.

Now it comes the time for the build pipeline. After creating it in Azure DevOps, you will need to select the repository which you linked to your ADF. Don't forget to use the appropriate agent. The Microsoft-hosted agent works just fine.

Then add tasks to the build pipeline. (if you want to start with a POC or a simple demo, you may just need an ARM template deployment.

To set up the Release Pipeline, you will need to add an artifact which will be the output of your build pipeline.

Then create stages for your environments (For example Dev, QA, Prod). Within each stage, add tasks to deploy your ADF pipelines, datasets...., using ARM templates.

For the automation part, you can set triggers in both your build and release pipelines. This way, whenever there's a change in the main branch of your repository, it will automatically initiate the build and, subsequently, the release.

Now, whenever you want to create or modify a pipeline, do it through the ADF UI (or the ADF SDK). Once you're happy with the changes, save and publish them. These changes will be saved to your Azure DevOps repository.

The CI/CD process you set up will automatically deploy your changes to the necessary environments.

Please test and tell us :)