Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
DatabricksUnityCatalog: notebooks lineage not showing up in table/view lineage or lineage graph
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(128, 8%, 22%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EA%3C/text%3E%3C/svg%3E)
Anonymous
Notebooks lineage not showing up in table&view lineage or lineage graph.
I created two table and one view from a notebook by following the doc: Capture and explore lineage
All lineages work fine, except the notebook lineage:
Lineage graph:
Notebook:
Environment: Azure DatabricksUnityCatalog, Premium (+ Role-based access controls), Lineage enabled, SQL Warehouse
Did I miss anything or do something wrong with it?
Azure Databricks
{count} votes
-
PRADEEPCHEEKATLA 91,831 Reputation points2023-11-12T13:36:44.8666667+00:00 @Anonymous - Thanks for the question and using MS Q&A platform.
Based on the information you provided, it seems like you have followed the correct steps to capture and explore lineage in Azure Databricks. However, sometimes it may take some time for the lineage to show up in the table/view lineage or lineage graph.
If you have waited for some time and the lineage is still not showing up, you can try the following steps:
- Check if the notebook is running and has completed successfully. If the notebook is not running or has failed, the lineage may not show up.
- Check if the tables and views have been created successfully. If there are any errors in creating the tables or views, the lineage may not show up.
- Check if the tables and views have been registered in the metastore. If the tables and views are not registered in the metastore, the lineage may not show up.
And also, according to official documentation, make sure followed all requirements to capture data lineage with Unity Catalog: Connect to and manage Azure Databricks Unity Catalog in Microsoft Purview (Preview)
If you have checked all of the above and the lineage is still not showing up, you may open a support ticket for further assistance.
-
Anonymous
2023-11-13T01:42:14.93+00:00 Thanks for your reply.
I think I have done the work you mentioned above. It seems the notebook lineage is just not showing up.
Every notebook run or schedule run displays as a new workflow lineage, the notebook lineage is still empty.
Maybe I misunderstood the documentations?
-
PRADEEPCHEEKATLA 91,831 Reputation points
2023-11-13T04:26:32.4933333+00:00 @Anonymous - Did you double checked all requirements to capture data lineage with Unity Catalog: Connect to and manage Azure Databricks Unity Catalog in Microsoft Purview (Preview).
If yes, if you have a support plan could you please file a support ticket for deeper investigation and do share the SR# with us?
-
Anonymous
2023-11-14T11:07:54.13+00:00 Thank you for your ideas @PRADEEPCHEEKATLA
I'm not using Purview to manage it, it's thexxx.azuredatabricks.netpages showing in the above screenshots.I will try to get support from support center
-
PRADEEPCHEEKATLA 91,831 Reputation points
2023-11-15T06:32:59.7433333+00:00 @Anonymous - Thanks for the update.
What happens when you click on See lineage graph?
-
Anonymous
2023-11-16T10:50:31.71+00:00 Click on
See lineage graphbutton then lineage graph shows up
, just like I pasted in this question content.
Thank you again
-
PRADEEPCHEEKATLA 91,831 Reputation points
2023-11-20T08:40:26.61+00:00 @Anonymous -I agree that this issue looks strange and I wasn't able to reproduce this issue. If you have a support plan could you please file a support ticket for deeper investigation and do share the SR# with us?
-
PRADEEPCHEEKATLA 91,831 Reputation points
2023-11-27T09:22:46.3233333+00:00 @Anonymous - Did you get a chance to open a support ticket?
-
Anonymous
2023-11-30T01:30:08.4+00:00 @PRADEEPCHEEKATLA-MSFT
Thank you for your concern.
I didn't find any option to open a support ticket in Azure Databricks portal. We set a new environment for development. After same notebook/same sample SQL run, lineages shows correctly in the new env. But notebook lineage still may not show up sometimes. Weird. -
PRADEEPCHEEKATLA 91,831 Reputation points
2023-12-01T10:03:56.5366667+00:00 @Anonymous - You can raise a support ticket from Azure Portal.
For more details, refer to Create an Azure support request.
Sign in to comment
Sign in to answer