Clarification on Specific Metadata Sync Between Databricks Unity Catalog and Microsoft Purview (Column Comments & Tags, Table Descriptions)

Question

Clarification on Specific Metadata Sync Between Databricks Unity Catalog and Microsoft Purview (Column Comments & Tags, Table Descriptions)

Mohammed Aamer 180

Hi Team,

I am currently working on a data governance initiative where we are scanning Azure Databricks Unity Catalog tables into Microsoft Purview and would like some clarification on metadata synchronization between the two systems.

Based on our understanding, the mapping between Databricks Unity Catalog and Microsoft Purview for the below metadata appears as follows (Ref. below screenshots):

Databricks Unity CatalogMicrosoft PurviewDescription (Table Level)Asset Description (Overview section - Table Level)Comment (Column Level)Column Description (Schema section - Column Level)Tags (Column Level)Tags (Schema section - Column Level)UC ss

file_000000003fb471f58f8da3b5f8327dfa

Purview ss

file_00000000b68c722f8da2ab4640dc0883

file_000000009608722f959bd792f5108898

Could you please confirm whether, during a Purview scan of Unity Catalog, either one or all of the above metadata fields are ingested into Purview?

If all are supported, please confirm.
If not all are supported, could you specify which of these fields are scanned and pushed from Databricks to Purview?
For any fields that are not being scanned/pushed from Databricks to Purview, is there an automated or recommended way (API-based, script-based, or native integration) to bring this metadata into Purview?
If no automated approach exists, what is the recommended best practice to maintain consistency between Databricks and Purview metadata?

Further, if Purview is able to fetch the corresponding metadata from Databricks for one or more of these fields, we have the below queries:

If a table or column comment/tag is updated in Databricks after an initial Purview scan, will the updated value from Databricks automatically reflect in Purview during subsequent scans?
If metadata such as Asset description, column description or tags is modified directly in Purview, will that update flow back to Databricks Unity Catalog, or will this create a mismatch between the two systems?
Lets say for initial Purview scan, it brings the column comment from ADB into Purview Column Description. Then we manually update the column description in Purview. Now when we do an incremental scan of Unity Catalog in Purview, will the updated column description in Purview be overwritten by the old/ original comment from ADB?
After a one-time scan, how does ongoing synchronization work?
- If purview has subsequent incremental scans, does this refresh metadata changes from Databricks?
Is the synchronization strictly one-way (Databricks → Purview), or is any bidirectional sync supported?

We are trying to determine the correct system of record and governance approach for maintaining table and column-level descriptions and tags across both platforms.

Please also let us know if there are any recommended best practices from Microsoft for managing metadata consistency between Databricks Unity Catalog and Microsoft Purview in production environments.

Regards,
Aamer

0 comments

Answer accepted by question author

0 additional answers

Your answer

Answer 1

Pilladi Padma Sai Manisha 11,715 Microsoft External Staff Moderator

Hi Mohammed Aamer,

Thank you for the detailed questions. I’ll clarify how metadata synchronization currently works between Azure Databricks Unity Catalog and Microsoft Purview.

The Unity Catalog scan in Purview performs metadata ingestion only and operates as a one-way synchronization from Databricks to Purview. There is no bidirectional metadata sync today.

During scans, Purview brings in technical metadata such as catalogs, schemas, tables, columns, and lineage. Table descriptions and column comments from Unity Catalog are supported and appear in Purview as asset description and column description. However, Unity Catalog tags are not fully synchronized as Purview schema tags, and native UC tags do not automatically map into Purview tags at this time.

When metadata is updated in Databricks, the changes will be reflected in Purview after the next scan, including incremental scans. If metadata such as descriptions or tags is edited directly in Purview, those updates do not flow back to Databricks because the integration is strictly one direction. Additionally, if the same metadata exists in Databricks, a future scan may overwrite values maintained only in Purview, which can lead to mismatches if both systems are edited independently.

The recommended best practice is to maintain technical metadata such as comments and descriptions in Unity Catalog as the system of record, and use Purview primarily for governance capabilities like classifications, glossary terms, ownership, and policy management. If synchronization of unsupported fields such as Unity Catalog tags is required, customers typically implement API-based automation using Purview APIs or Databricks metadata queries, as there is currently no native automated sync for those fields.

References:
Register & scan Azure Databricks Unity Catalog in Microsoft Purview
Unity Catalog comments and metadata documentation

Mohammed Aamer 180 Reputation points

2026-02-19T15:52:36.39+00:00
Hi,

Thank you for the clarification, I have a few follow-up questions:

1. Purview API implementation (for UC tags):

What is the recommended approach or documentation for implementing Purview APIs to sync metadata (e.g., tags)?

Are there any official samples/reference architectures available?

Are Purview APIs included within standard Purview licensing, or is there any additional cost?

2. Automation:

If we implement API-based synchronization, do we need to schedule and automate the API execution ourselves (e.g., via ADF/Azure Functions)? Is there any recommended orchestration pattern?

3. Overwrite behavior (Unity Catalog → Purview):

Currently, we do not have access to update column comments in Databricks Unity Catalog, so those fields are empty in Unity Catalog. We are therefore maintaining column descriptions directly in Purview.

In this scenario, we would like clarification on scan behavior:

If the column comment in Unity Catalog is empty but the column description is populated in Purview, during an incremental scan will Purview retain its existing description or overwrite it with a blank value from Unity Catalog? In other words, does Unity Catalog always act as the source of truth during scans even when the corresponding metadata field is empty?

In the future, if column comments are later added in Unity Catalog for those same columns which already have been manually maintained in Purview, will a subsequent scan overwrite the existing Purview column descriptions with the Unity Catalog comments?

This will help us determine how to manage Purview descriptions until Unity Catalog comments become available.

Regards,

Aamer
Mohammed Aamer 180 Reputation points

2026-02-20T18:21:08.1866667+00:00

Hi @Pilladi Padma Sai Manisha ,

Could you please help me on the above queries.
Pilladi Padma Sai Manisha 11,715 Reputation points Microsoft External Staff Moderator

2026-02-24T16:51:16.1066667+00:00

Hi Mohammed Aamer
Sorry for delay in response
Regarding Purview APIs, you can use the Microsoft Purview Catalog (Atlas) REST APIs to programmatically update metadata such as descriptions, classifications, or custom attributes. There is currently no native or Microsoft-provided reference architecture specifically for syncing Unity Catalog tags, so customers typically build a custom integration that reads metadata from Databricks and pushes updates into Purview. These APIs are included as part of standard Purview capabilities, and there is no separate licensing cost beyond normal Purview usage.

For automation, API execution needs to be scheduled and managed by your side. Common orchestration patterns include Azure Functions, Azure Data Factory, or Logic Apps to periodically sync metadata after scans. The Unity Catalog connector itself does not provide built-in automation for this.

Regarding overwrite behavior, Unity Catalog should be treated as the upstream source when metadata exists there. If column comments are empty in Unity Catalog and descriptions exist only in Purview, incremental scans typically retain the Purview value because no source update is provided, although this isn’t guaranteed behavior. If comments are later added in Unity Catalog, future scans are expected to update or overwrite the Purview column descriptions with the Unity Catalog values.

Best practice is to use Unity Catalog as the system of record for technical metadata once it becomes available, and use Purview mainly for governance enrichment.
Pilladi Padma Sai Manisha 11,715 Reputation points Microsoft External Staff Moderator

2026-02-27T09:43:08.39+00:00

Hi Mohammed Aamer,
I hope you had a chance to review the information shared earlier, and I hope this information has been helpful! If you still have questions, please let us know what is needed in the comments so the question can be answered.