Push Data Factory lineage data to Microsoft Purview

Article
08/10/2023

APPLIES TO: Azure Data Factory Azure Synapse Analytics

Tip

Try out Data Factory in Microsoft Fabric, an all-in-one analytics solution for enterprises. Microsoft Fabric covers everything from data movement to data science, real-time analytics, business intelligence, and reporting. Learn how to start a new trial for free!

In this tutorial, you'll use the Data Factory user interface (UI) to create a pipeline that run activities and report lineage data to Microsoft Purview account. Then you can view all the lineage information in your Microsoft Purview account.

Currently, lineage is supported for Copy, Data Flow, and Execute SSIS activities. Learn more details on the supported capabilities from Supported Azure Data Factory activities.

Prerequisites

Azure subscription. If you don't have an Azure subscription, create a free Azure account before you begin.
Azure Data Factory. If you don't have an Azure Data Factory, see Create an Azure Data Factory.
Microsoft Purview account. The Microsoft Purview account captures all lineage data generated by data factory. If you don't have a Microsoft Purview account, see Create a Microsoft Purview.

Run pipeline and push lineage data to Microsoft Purview

Step 1: Connect Data Factory to your Microsoft Purview account

You can establish the connection between Data Factory and Microsoft Purview account by following the steps in Connect Data Factory to Microsoft Purview.

Step 2: Run pipeline in Data Factory

You can create pipelines, Copy activities and Dataflow activities in Data Factory. You don't need any additional configuration for lineage data capture. The lineage data will automatically be captured during the activities execution.

Screenshot of Copy and Dataflow activity.

Screenshot of Execute SSIS Package activity.

Learn more about how to create Copy, Data Flow and Execute SSIS activities from Copy data from Azure Blob storage to a database in Azure SQL Database by using Azure Data Factory, Transform data using mapping data flows and Run SSIS Packages in Azure.

Step 3: Monitor lineage reporting status

After you run the pipeline, in the pipeline monitoring view, you can check the lineage reporting status by clicking the following Lineage status button. The same information is also available in the activity output JSON -> reportLineageToPurvew section.

Monitor the lineage reporting status in pipeline monitoring view.

Step 4: View lineage information in your Microsoft Purview account

On Microsoft Purview UI, you can browse assets and choose type "Azure Data Factory". You can also search the Data Catalog using keywords.

On the activity asset, click the Lineage tab, you can see all the lineage information.

Copy activity:
Data Flow activity:

Note

For the lineage of Dataflow activity, we only support source and sink. The lineage for Dataflow transformation is not supported yet.
Execute SSIS Package activity:

Note

For the lineage of Execute SSIS Package activity, we only support source and destination. The lineage for transformation is not supported yet.