Query on Databricks Notebook IDs in Purview Lineage

Question

Query on Databricks Notebook IDs in Purview Lineage

Mohammed Aamer 160

Hi Team,

I am working on Microsoft Purview to scan Azure Databricks Unity Catalog for table and column lineage. In our project, Databricks notebooks are used for transformations and ADF is used for orchestration. We have 15 Databricks notebooks handling all transformations, and the jobs have been running in production for several months. We are performing incremental scans in Purview (not fresh scans).

While reviewing lineage in Purview, I noticed that although the same notebooks are used, there are ~200+ notebook IDs appearing in lineage.

From a data governance perspective, we want to add asset descriptions to these notebook nodes. However, if the notebook ID keeps changing, the descriptions entered earlier become obsolete.

Could you please clarify:

Is our observation correct that a new notebook ID is generated for each run/execution of a Databricks notebook in lineage?
If yes, is there any way to map or anchor these run-based notebook IDs to a stable notebook asset description (e.g., notebook path/name) so governance metadata like descriptions can be maintained consistently?

Please see below screenshots for your reference.
Lineage

Asset Desc

Regards,
Aamer

Answer accepted by question author

0 additional answers

Your answer

Answer 1

Smaran Thoomu 33,840 Microsoft External Staff Moderator

Hi Mohammed Aamer
Good observation - what you’re seeing is expected with how Databricks lineage is captured today.

In Purview, the notebook nodes that appear in lineage are often tied to the execution/run context, not just the logical notebook file. So when the same notebook runs multiple times (for example through ADF orchestration), Purview can show multiple notebook assets with different IDs. These IDs are system-generated and can look like new notebooks even though the source notebook is the same.

So yes, it can result in many notebook IDs over time, especially in active production environments.

Right now, there isn’t a way to “anchor” all those run-based IDs to one stable notebook asset in Purview. The lineage is reflecting execution history rather than a single static notebook object.

For governance, most teams handle this by:

Using the notebook path/name in Databricks as the main reference
Adding descriptions at the table or data asset level instead of each notebook node
Treating notebook nodes more as technical lineage artifacts than governed assets

This is a current product behavior rather than a scan issue, and incremental vs full scans don’t change it much.

If Microsoft improves notebook normalization in lineage in the future, this should become easier to manage.

Hope this clarifies what you’re seeing.

Mohammed Aamer 160 Reputation points

2026-02-17T08:58:21.7066667+00:00
Hi Smaran,

Thank you, this is very helpful. I have a few follow-up questions regarding the governance approach you suggested:

Using the notebook path/name in Databricks as the main reference

Do you mean that we should manually maintain a mapping of which notebook creates or updates which tables/views (based on notebook path/name)?

Adding descriptions at the table/view asset level instead of notebook nodes

From an industry best-practice perspective, what details are typically included in table/view asset descriptions for Databricks notebook?

Is there any recommended standard template (for example, should upstream and downstream notebook names be included)?

Also, where should these descriptions ideally be maintained - in Microsoft Purview under the table/view asset description (as shown in the screenshot), or elsewhere?

We currently have ~800 tables/views, so adding detailed Databricks notebook-related descriptions manually for each asset would be a substantial effort. Any scalable or recommended approach for updating these details in Purview would be very helpful.

Regards,
Aamer
Smaran Thoomu 33,840 Reputation points Microsoft External Staff Moderator

2026-02-18T11:29:42.5866667+00:00
Hi Mohammed Aamer
Good follow-up questions.

Notebook path/name as reference

Yes, since Purview notebook IDs are run-based, most teams use the Databricks notebook path/name as the stable reference. This mapping is usually kept in internal docs or in the Databricks workspace itself, not in Purview.

Purview lineage is more for visibility of flow, not for maintaining notebook ownership details.

What to put in table/view descriptions

In practice, table descriptions are kept business-focused. Common things to include:

What the table is for

Where the data mainly comes from

How often it’s refreshed

Who owns it

Including notebook names in every table description is usually avoided because it’s hard to maintain.

Where to maintain it

Purview is a good place for business descriptions (like in your screenshot). Detailed notebook-to-table mapping is often kept in project docs or a team wiki.

For large numbers of tables (~800)

Updating one by one isn’t realistic. Teams usually:

Focus on key/curated tables first

Use APIs for bulk updates if needed

Document critical datasets more deeply than raw ones

That’s the practical approach most customers follow.

Hope this helps.

Share via

Query on Databricks Notebook IDs in Purview Lineage

0 additional answers

Your answer