How to create Data Lineage for given scenario

Naren Babu 20

Hi All, I am trying to create a data lineage in Microsoft purview.

The data lineage is for an attribute whose origin and destination is as below:

SQL server --> Azure Data Factory --> Azure Data Lake Storage Gen2 --> Azure Databricks --> PYSPARK --> SPARK SQL --> Microsoft Power BI

How to build a lineage in MS purview for attribute that is passing through each of these processes and steps?

phemanth 10,555 Reputation points Microsoft Vendor

2024-08-13T16:12:44.94+00:00
@Naren Babu

Thanks for using MS Q&A platform and posting your query.

Creating a data lineage in Microsoft Purview for your specified data flow involves several steps.

Here’s the steps to help you get started:

Set Up Microsoft Purview Account:

Ensure you have a Microsoft Purview account set up and configured.

Connect Data Sources:

SQL Server: Connect your SQL Server to Microsoft Purview to start capturing metadata and lineage.

Azure Data Factory: Integrate Azure Data Factory with Purview to capture lineage from data movement activities like Copy and Data Flow activities.

Configure Data Storage:

Azure Data Lake Storage Gen2: Connect your ADLS Gen2 to Purview to capture lineage from data stored and processed in the lake.

Integrate Data Processing:

Azure Databricks: Connect Databricks to Purview to capture lineage from notebooks and jobs executed in Databricks.

PYSPARK and SPARK SQL: Ensure that the transformations and queries executed in PySpark and Spark SQL are captured by Databricks integration.

Connect Analytics and Visualization:

Microsoft Power BI: Integrate Power BI with Purview to capture lineage from datasets and reports created in Power Bi.

Manual Lineage Addition:

For any steps or transformations not automatically captured, you can manually add lineage in Purview. Navigate to the asset in the data catalog, select the Lineage tab, and add manual lineage.

Verify and Visualize Lineage:

Once all connections are established, verify the lineage in the Purview Data Catalog. You should see a visual representation of the data flow from SQL Server through to Power BI.

for more details, please refer below links

https://learn.microsoft.com/en-us/purview/catalog-lineage-user-guide

https://learn.microsoft.com/en-us/purview/catalog-lineage-user-guide

By following these steps, you can build a comprehensive data lineage in Microsoft Purview that captures the entire journey of your data attribute.

Hope this helps. Do let us know if you any further queries.
phemanth 10,555 Reputation points Microsoft Vendor

2024-08-14T11:50:46.3466667+00:00

@Naren Babu We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
phemanth 10,555 Reputation points Microsoft Vendor

2024-08-16T15:03:53.6+00:00
@Naren Babu

To better understand the issue, can you please provide more details about the lineage you got from Purview and how it differs from the actual data flow you implemented?

Here are some questions to help me better understand the situation:

What does the lineage from Purview show? Does it show the data flow from the API to Power BI, but with some missing or incorrect steps?

Are there any specific steps or components in your implementation that are not represented in the lineage?

Have you checked the metadata and lineage settings in Purview to ensure that all the necessary data sources and processing steps are properly configured and connected?

please do check the below steps and confirm us

1. Review Purview connections and metadata:

Ensure that all data sources, including the API, Lakehouse, and Delta tables, are properly connected to Purview.

Verify that the metadata for each component is accurate and up to date.

2. Check Data Factory pipeline configuration:

Review the Data Factory pipeline configuration to ensure that it is correctly capturing the lineage from the API to the Lakehouse and subsequent processing steps.

Check if the pipeline is configured to capture metadata and lineage for each activity, including the Python code and notebook execution.

3. Verify Python code and notebook execution:

Ensure that the Python code and notebook execution are properly configured to capture metadata and lineage.

Check if the Python code is using any libraries or frameworks that might not be compatible with Purview's lineage capture.

4. Manually add lineage in Purview (if necessary):

If some steps or components are not automatically captured, you can manually add lineage in Purview.

Navigate to the asset in the data catalog, select the Lineage tab, and add manual lineage to fill in the gaps.

5. Re-run the pipeline and refresh Purview:

Re-run the Data Factory pipeline to ensure that the latest metadata and lineage are captured.

Refresh the Purview data catalog to reflect the updated lineage.
phemanth 10,555 Reputation points Microsoft Vendor

2024-08-19T16:25:48.5733333+00:00

@Naren Babu We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
Naren Babu 20 Reputation points

2024-08-19T17:11:39.9+00:00

Will update in next one day about details.
phemanth 10,555 Reputation points Microsoft Vendor

2024-08-23T11:09:53.32+00:00

@Naren Babu was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.

1 answer

Naren Babu 20 Reputation points

2024-08-14T12:20:23.98+00:00

Hi, I have a different scenario now.. the previous one that i mentioned above is yet to be implemented. However, I have implemented a scenario in Microsoft Fabric:

Data factory Pipeline (API for bing news search --> Load into Lakehouse --> python code to extract json and store in dataframe --> python notebook to create sentiment analysis and load into another delta table) --> Power BI Report .

I got a lineage from Purview, but that doesn't represent exactly the way i explained above.

So wondering how to fix this.. any way to suggest
Please sign in to rate this answer.

0 comments No comments
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

Share via

How to create Data Lineage for given scenario

1 answer

Your answer