How to create documentation on existing ADF with lot of pipelines?

Question

We have inherited a project which has 300 plus pipelines built in azure data factory without any documentation. Therefore, we are trying to create documentations on ADF with details such as PipelineName, Sources (Linked Services, Datasets), Destination (Linked Services, Datasets), Folder Name of pipeline, schedule, Activity list, description of pipeline, etc in one single place related to each other. We are able to get data in a list individually, but we need combined together relating to each column. Tried to API and CLI, not luck. Any help is much appreciated.

Answer 1

The trigger definition contains reference to pipelines. The pipelines do not contain reference to trigger.

This one-directionality is so additional pipelines can be added to a trigger without re-publishing the pipeline. Indeed it is the trigger which calls the pipeline, not the pipeline calling the trigger.

In pipeline definition, each activity which uses a dataset with have, burried somewhere inside) a subobject like:

{  
     "referenceName": "MyAzureBlobDataset",  
     "type": "DatasetReference"  
 }  

With such a pattern, we could write a regex expression to extract each instance from the pipeline definitions. I think the regex might be easier than trying to navigate all the JSON.

Does this help @Srikanth Vunyala ?

Answer 2

Here is link to Microsoft documentation for connecting to Azure Data Factory and listing pipelines.

https://learn.microsoft.com/en-us/rest/api/datafactory/pipelines/list-by-factory?tabs=HTTP

Answer 3

Hi @Srikanth Vunyala

Perhaps the following guide could assist:

https://www.techtalkcorner.com/documenting-azure-data-factory/

Scripts can be found on:

https://github.com/techtalkcorner/AzDataFactory/tree/master/Scripts/PowerShell

-----------------------------

If this is helpful please accept answer.