Purview not generating lineage for Azure data factory pipelines (Data flow, Copydata)

Question

Purview not generating lineage for Azure data factory pipelines (Data flow, Copydata)

Hanuman Devineni 51

Unable to view lineage for ADF copy data and data flow in the purview of adding ADF & Scanning the database.

Had a private endpoint configured prior and tried removing and checking but no success.

PRADEEPCHEEKATLA 90,651 Reputation points Moderator

2024-07-24T02:21:26.1933333+00:00

@Hanuman Devineni - Thanks for the question and using MS Q&A platform.

I'm sorry to hear that you're having trouble viewing lineage for ADF copy data and data flow in Microsoft Purview. Based on the document I have, Microsoft Purview drops lineage if the source or destination uses an unsupported data storage system. Supported data sources in copy activity are listed in the Copy activity support of Connect to Azure Data Factory - Copy activity support. Similarly, supported data sources in data flow activity are listed in the Data Flow support of Connect to Azure Data Factory - Data Flow support.

If your data sources are supported, then there might be some limitations on copy activity lineage that could be causing the issue. You can find these limitations in the Limitations on copy activity lineage section of the same document.

If you have already checked these documents and are still unable to view lineage, it might be helpful to provide more information about your specific scenario. Could you please provide more details about your configuration and the steps you have taken so far? This will help me better understand your situation and provide more specific guidance.

Looking forward for your response.
Hanuman Devineni 51 Reputation points

2024-07-24T03:11:31.43+00:00

Thanks for quick response, Source is SQL server and target is SQL MI instance so I believe both are supported, We are looking at copy data and data flow using individual pipeline for each table, suggest us the process to review on why lineage is not captured.
PRADEEPCHEEKATLA 90,651 Reputation points Moderator

2024-07-24T10:03:50.7066667+00:00

@Hanuman Devineni - Thank you for providing more information. Since your source and target are both supported, we can look into some other possible reasons why lineage is not being captured.

First, please make sure that you have followed the step-by-step guide to push Azure Data Factory lineage in Microsoft Purview. This guide provides detailed instructions on how to configure your Azure Data Factory and Microsoft Purview accounts to enable lineage tracking.

If you have followed these steps and are still unable to view lineage, there might be some issues with the configuration of your private endpoints. Please refer to the Access a secured Microsoft Purview account document for more information on how to set up private endpoints for Microsoft Purview.

If you have already checked these documents and are still unable to view lineage, it might be helpful to check the logs for your pipeline runs to see if there are any errors or warnings related to lineage tracking. You can find these logs in the Azure portal under the "Monitor" tab for your pipeline runs.

If you are still unable to resolve the issue, please provide more details about your specific configuration and any error messages you are seeing. This will help me better understand your situation and provide more specific guidance. Looking forward for your response.
PRADEEPCHEEKATLA 90,651 Reputation points Moderator

2024-07-29T06:51:30.2066667+00:00

@Hanuman Devineni - We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.

Your answer

PRADEEPCHEEKATLA 90,651 Reputation points Moderator

2024-07-24T02:21:26.1933333+00:00

@Hanuman Devineni - Thanks for the question and using MS Q&A platform.

I'm sorry to hear that you're having trouble viewing lineage for ADF copy data and data flow in Microsoft Purview. Based on the document I have, Microsoft Purview drops lineage if the source or destination uses an unsupported data storage system. Supported data sources in copy activity are listed in the Copy activity support of Connect to Azure Data Factory - Copy activity support. Similarly, supported data sources in data flow activity are listed in the Data Flow support of Connect to Azure Data Factory - Data Flow support.

If your data sources are supported, then there might be some limitations on copy activity lineage that could be causing the issue. You can find these limitations in the Limitations on copy activity lineage section of the same document.

If you have already checked these documents and are still unable to view lineage, it might be helpful to provide more information about your specific scenario. Could you please provide more details about your configuration and the steps you have taken so far? This will help me better understand your situation and provide more specific guidance.

Looking forward for your response.
Hanuman Devineni 51 Reputation points

2024-07-24T03:11:31.43+00:00

Thanks for quick response, Source is SQL server and target is SQL MI instance so I believe both are supported, We are looking at copy data and data flow using individual pipeline for each table, suggest us the process to review on why lineage is not captured.
PRADEEPCHEEKATLA 90,651 Reputation points Moderator

2024-07-24T10:03:50.7066667+00:00

@Hanuman Devineni - Thank you for providing more information. Since your source and target are both supported, we can look into some other possible reasons why lineage is not being captured.

First, please make sure that you have followed the step-by-step guide to push Azure Data Factory lineage in Microsoft Purview. This guide provides detailed instructions on how to configure your Azure Data Factory and Microsoft Purview accounts to enable lineage tracking.

If you have followed these steps and are still unable to view lineage, there might be some issues with the configuration of your private endpoints. Please refer to the Access a secured Microsoft Purview account document for more information on how to set up private endpoints for Microsoft Purview.

If you have already checked these documents and are still unable to view lineage, it might be helpful to check the logs for your pipeline runs to see if there are any errors or warnings related to lineage tracking. You can find these logs in the Azure portal under the "Monitor" tab for your pipeline runs.

If you are still unable to resolve the issue, please provide more details about your specific configuration and any error messages you are seeing. This will help me better understand your situation and provide more specific guidance. Looking forward for your response.
PRADEEPCHEEKATLA 90,651 Reputation points Moderator

2024-07-29T06:51:30.2066667+00:00

@Hanuman Devineni - We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.

Share via

Purview not generating lineage for Azure data factory pipelines (Data flow, Copydata)

Your answer