Notebooks lineage not showing up in table&view lineage or lineage graph. I created two table and one view from a notebook by following the doc: Capture and explore lineage All lineages work fine, except the notebook lineage: Lineage graph: Notebook: Environment: Azure DatabricksUnityCatalog, Premium (+ Role-based access controls), Lineage enabled, SQL Warehouse Did I miss anything or do something wrong with it?

DatabricksUnityCatalog: notebooks lineage not showing up in table/view lineage or lineage graph

Mikes 0 Microsoft Vendor

Notebooks lineage not showing up in table&view lineage or lineage graph.

I created two table and one view from a notebook by following the doc: Capture and explore lineage

All lineages work fine, except the notebook lineage:

User's image

Lineage graph:

User's image

Notebook:

User's image

Environment: Azure DatabricksUnityCatalog, Premium (+ Role-based access controls), Lineage enabled, SQL Warehouse

Did I miss anything or do something wrong with it?

PRADEEPCHEEKATLA-MSFT 85,746 Reputation points Microsoft Employee

2023-11-12T13:36:44.8666667+00:00
@Mikes - Thanks for the question and using MS Q&A platform.

Based on the information you provided, it seems like you have followed the correct steps to capture and explore lineage in Azure Databricks. However, sometimes it may take some time for the lineage to show up in the table/view lineage or lineage graph.

If you have waited for some time and the lineage is still not showing up, you can try the following steps:

Check if the notebook is running and has completed successfully. If the notebook is not running or has failed, the lineage may not show up.

Check if the tables and views have been created successfully. If there are any errors in creating the tables or views, the lineage may not show up.

Check if the tables and views have been registered in the metastore. If the tables and views are not registered in the metastore, the lineage may not show up.

And also, according to official documentation, make sure followed all requirements to capture data lineage with Unity Catalog: Connect to and manage Azure Databricks Unity Catalog in Microsoft Purview (Preview)

If you have checked all of the above and the lineage is still not showing up, you may open a support ticket for further assistance.
Mikes 0 Reputation points Microsoft Vendor

2023-11-13T01:42:14.93+00:00

@PRADEEPCHEEKATLA-MSFT

Thanks for your reply.

I think I have done the work you mentioned above. It seems the notebook lineage is just not showing up.

Every notebook run or schedule run displays as a new workflow lineage, the notebook lineage is still empty.

Maybe I misunderstood the documentations?
PRADEEPCHEEKATLA-MSFT 85,746 Reputation points Microsoft Employee

2023-11-13T04:26:32.4933333+00:00

@Mikes - Did you double checked all requirements to capture data lineage with Unity Catalog: Connect to and manage Azure Databricks Unity Catalog in Microsoft Purview (Preview).

If yes, if you have a support plan could you please file a support ticket for deeper investigation and do share the SR# with us?
Mikes 0 Reputation points Microsoft Vendor

2023-11-14T11:07:54.13+00:00

Thank you for your ideas @PRADEEPCHEEKATLA-MSFT
I'm not using Purview to manage it, it's the xxx.azuredatabricks.net pages showing in the above screenshots.

I will try to get support from support center
PRADEEPCHEEKATLA-MSFT 85,746 Reputation points Microsoft Employee

2023-11-15T06:32:59.7433333+00:00

@Mikes - Thanks for the update.

What happens when you click on See lineage graph?
Mikes 0 Reputation points Microsoft Vendor

2023-11-16T10:50:31.71+00:00

Click on See lineage graph button then lineage graph shows up

, just like I pasted in this question content.

Thank you again
PRADEEPCHEEKATLA-MSFT 85,746 Reputation points Microsoft Employee

2023-11-20T08:40:26.61+00:00

@Mikes -I agree that this issue looks strange and I wasn't able to reproduce this issue. If you have a support plan could you please file a support ticket for deeper investigation and do share the SR# with us?
PRADEEPCHEEKATLA-MSFT 85,746 Reputation points Microsoft Employee

2023-11-27T09:22:46.3233333+00:00

@Mikes - Did you get a chance to open a support ticket?
Mikes 0 Reputation points Microsoft Vendor

2023-11-30T01:30:08.4+00:00

@PRADEEPCHEEKATLA-MSFT
Thank you for your concern.
I didn't find any option to open a support ticket in Azure Databricks portal. We set a new environment for development. After same notebook/same sample SQL run, lineages shows correctly in the new env. But notebook lineage still may not show up sometimes. Weird.
PRADEEPCHEEKATLA-MSFT 85,746 Reputation points Microsoft Employee

2023-12-01T10:03:56.5366667+00:00

@Mikes - You can raise a support ticket from Azure Portal.

For more details, refer to Create an Azure support request.

Share via

DatabricksUnityCatalog: notebooks lineage not showing up in table/view lineage or lineage graph