Copy Data Lineage Activity is not reported in Purview

Question

Hi,

Context: I have a data pipeline in ADF. The pipeline contains the following activities

• A total of 15 number of copy data activity
• Total 1 number of Data Flow activity
• The data source for all activities is a common blob storage account.
• The data sink for all activities is a different but common blob storage account.

The data source and data sink are registered in Purview. Data Factory is also successfully connected with Microsoft Purview. Data Factory managed identity has the Data Curator access at the root collection level in Purview. There is no firewall neither any network issue exists between ADF and Purview.

Problem: Out of all activities only the Data Flow activity Lineage is reported in Purview, whereas the Lineage status for all Copy Data activity shows the error: Status: Timed Out, Failed to report lineage to catalog in 00:02:00.

Please help me figure out how to fix this issue.

Answer 1

@Ravineesh - Thanks for the question and using MS Q&A platform.

Could you please confirm whether you had created private endpoint in purview to connect the data sources and disable public network access?

If yes, please do make sure copy activity or data flow activity in ADF/Synapse is running via Managed Vnet IR.

Note: Please be aware that Azure IR or AutoResolvedIntegationRuntime is designed to make outbound connection via public network. So, it will not work in this scenario.

If no, please check if there are any issues with the integration runtime used to execute the Copy Data activities and conduct lineage push. You can check the integration runtime logs to see if there are any errors or warnings related to the lineage push. If there are any issues with the integration runtime, you may need to troubleshoot and fix them.

Hope this helps. Do let us know if you any further queries.