Push Data Factory lineage data to Microsoft Purview
APPLIES TO: Azure Data Factory Azure Synapse Analytics
Tip
Try out Data Factory in Microsoft Fabric, an all-in-one analytics solution for enterprises. Microsoft Fabric covers everything from data movement to data science, real-time analytics, business intelligence, and reporting. Learn how to start a new trial for free!
In this tutorial, you'll use the Data Factory user interface (UI) to create a pipeline that run activities and report lineage data to Microsoft Purview account. Then you can view all the lineage information in your Microsoft Purview account.
Currently, lineage is supported for Copy, Data Flow, and Execute SSIS activities. Learn more details on the supported capabilities from Supported Azure Data Factory activities.
Prerequisites
- Azure subscription. If you don't have an Azure subscription, create a free Azure account before you begin.
- Azure Data Factory. If you don't have an Azure Data Factory, see Create an Azure Data Factory.
- Microsoft Purview account. The Microsoft Purview account captures all lineage data generated by data factory. If you don't have a Microsoft Purview account, see Create a Microsoft Purview.
Run pipeline and push lineage data to Microsoft Purview
Step 1: Connect Data Factory to your Microsoft Purview account
You can establish the connection between Data Factory and Microsoft Purview account by following the steps in Connect Data Factory to Microsoft Purview.
Step 2: Run pipeline in Data Factory
You can create pipelines, Copy activities and Dataflow activities in Data Factory. You don't need any additional configuration for lineage data capture. The lineage data will automatically be captured during the activities execution.
Learn more about how to create Copy, Data Flow and Execute SSIS activities from Copy data from Azure Blob storage to a database in Azure SQL Database by using Azure Data Factory, Transform data using mapping data flows and Run SSIS Packages in Azure.
Step 3: Monitor lineage reporting status
After you run the pipeline, in the pipeline monitoring view, you can check the lineage reporting status by clicking the following Lineage status button. The same information is also available in the activity output JSON -> reportLineageToPurvew
section.
Step 4: View lineage information in your Microsoft Purview account
On Microsoft Purview UI, you can browse assets and choose type "Azure Data Factory". You can also search the Data Catalog using keywords.
On the activity asset, click the Lineage tab, you can see all the lineage information.
Copy activity:
Data Flow activity:
Note
For the lineage of Dataflow activity, we only support source and sink. The lineage for Dataflow transformation is not supported yet.
Execute SSIS Package activity:
Note
For the lineage of Execute SSIS Package activity, we only support source and destination. The lineage for transformation is not supported yet.