Purview Databricks Notebook Scan get ID instead of name

Question

Dear Microsoft Support, I have successfully using Azure Purview to Scan Databricks Unity Catalog and get lineage correctly, however I get only Notebook ID, is it possible to put Notebook Name to it instead of ID ? It is not meaningful to have Notebook ID at all instead of Notebook Name

Answer 1

Hi @Vorapol Santhadkolkarn

I understand that you're successfully scanning your Databricks Unity Catalog with Microsoft Purview and capturing lineage, but you're seeing Notebook IDs instead of Notebook Names, which makes it less meaningful for your use case.

Currently, Microsoft Purview captures lineage information from Azure Databricks Unity Catalog primarily using identifiers like Notebook IDs, as these are unique and directly reference the notebook within the Databricks environment. The Notebook Name, while more human-readable, isn't natively included in the lineage metadata that Purview pulls from Unity Catalog's system tables or APIs.

Why Notebook IDs Are Used

Notebook IDs are likely used because they are immutable and uniquely identify a notebook, even if its name changes over time. Names, on the other hand, can be edited by users, potentially leading to ambiguity in lineage tracking. However, I completely agree that for usability, seeing the Notebook Name would be far more intuitive.

I hope this information helps. Please do let us know if you have any further queries.

Kindly consider upvoting the comment if the information provided is helpful. This can assist other community members in resolving similar issues.

Thank you.

Answer 2

IDs - I cannot find any of this IDs in Databricks - so effectively I cannot find which Notebook is mentioned :(