Clarification on Custom Classification Auto-Tagging in Microsoft Purview

Question

Hi Team,

We are currently scanning Azure Databricks Unity Catalog tables into Microsoft Purview as part of our data governance implementation.

In Purview, under each table’s Schema section, we can see that certain columns are automatically assigned built-in classifications (for example, “All Full Names”) based on Microsoft’s predefined classification rules. See below screenshot for reference.

We would like to confirm the behavior for custom classifications:

If we create custom classification rules in Microsoft Purview based on our own logic and patterns, will Purview automatically apply those classifications to all applicable columns across the Databricks tables?

Specifically:

1. After creating custom classifications, if we perform an incremental scan of the ADB Unity Catalog in Purview, will it automatically apply those custom classifications to matching columns in already scanned tables?
2. Is a re-scan mandatory for the custom classifications to take effect on existing assets?
3. Will these custom classifications also be applied automatically to new tables/columns in future scans?
4. If automatic tagging does not occur, will we need to manually assign classifications to each column individually?

Please confirm the expected behavior and any recommended configuration steps.

Regards,
Aamer

Answer 1

Hi Mohammed Aamer,

It sounds like you’re diving deep into custom classifications in Microsoft Purview with Azure Databricks. Here’s what I can share based on your queries:

Automatic Application of Custom Classifications:

• If you create custom classification rules, Purview will not automatically apply those classifications to matching columns in previously scanned tables during an incremental scan.

Re-scan Requirements:

  - Yes, a **re-scan is mandatory** for the custom classifications to take effect on existing assets. This is essential to ensure that the new rules are considered.
  
  **Future Scans and New Tables/Columns**:
  
     - Custom classifications **will be applied automatically** to any new tables or columns that match your rules in future scans. Just make sure your classifications are correctly configured.
     
     **Manual Assignment**:
     
        - If automatic tagging doesn’t occur, you will indeed need to **manually assign** classifications to each column individually for existing assets.
        

Recommended Steps:

• Make sure your custom classifications meet the data pattern and column pattern requirements.
• Regularly perform scans on your Unity Catalog to ensure all new assets are correctly classified.

I hope this helps clarify the behavior you're experiencing with Microsoft Purview! If you have any more questions or need further assistance, feel free to ask.

References:

• Custom classifications
• Classification best practices
• System classifications