Seperate repositories for type of ADF pipelines - Recommendation ?

Question

to-date, all pipelines developed have been using ADF without a devops github repository format. Now i need to do that. This is a new concept for me, so i was thinking of having multiple repositories for types of pipelines

ex: All pipelines that load from source A to db is one repository
Another pipeline that has a completely different purpose into a different repository.

Is that approach recommended or just have all pipelines into a single repository ?

Answer 1

Hi @sam nick ,

Yes, I think it's good approach to have separate ADFs for different projects/repositories for different set of pipelines. You can also setup a CI/CD Pipeline for your ADF projects. This will help you to manage separate ADF environments (development, test, production). Please refer to this documentation for more details.

Please "Accept the answer" if this helps or let me know more details about your scenario.

More references:
best-practices-to-implement-an-azure-data-factory
azure-data-factory-3-ways-to-integrate-adf-pipeline-with-azure-devops-git

Answer 2

That's not a recommended approach.

In a data factory, you have 5 pipelines. You can't link these 5 individual pipelines to 5 individual repositories as far as I know.

Ideal way is to create a development data factory, link it to a repository and all the 5 pipelines, related datasets and linked services code gets saved to that single linked repository.

Official document here: https://learn.microsoft.com/en-us/azure/data-factory/continuous-integration-delivery#cicd-lifecycle

----------

Please don't forget to Accept Answer and Up-vote if the response helped -- Vaibhav