Purview Scans are creating Schema Classifications but not Data Classifications

Question

Purview Scans are creating Schema Classifications but not Data Classifications

Jarrod Henningsen 20

Full scans have been run on an ADLS Gen2 storage account containing an example csv file with PII data at both the Auto and Level3 ScanLevel. Schema Classifications were found for PII related rules such as physical address, full name, etc, but no classifications were created. Scan rule set AdlsGen2 (System) was used for the scan. Why are Data Classifications not being generated.

Anonymous

2025-02-27T20:09:11.9366667+00:00
@Jarrod Henningsen

Welcome to the Microsoft Q&A and thank you for posting your questions here.

When using Azure Purview to scan data in an ADLS Gen2 storage account, it's important to understand the distinction between schema classifications and data classifications. Schema classifications identify the structure of the data (like the types of fields in a CSV), while data classifications focus on the actual content of the data.

Here are troubleshooting steps; please check and confirm us whether the issue gets resolved:

Ensure that the scan rule set you are using (AdlsGen2 (System)) is configured to detect and classify the specific types of data you are expecting. Sometimes, custom rule sets might be needed.

Verify that the classification rules for data classifications are correctly defined and enabled. Schema classifications might be working because they are predefined, but data classifications might need additional configuration.

Check if the data sensitivity labels are properly applied and recognized by the system. Sometimes, the absence of these labels can prevent data classifications from being generated.

Ensure that the scanning service has the necessary permissions to access and classify the data within the ADLS Gen2 storage account.

Although you mentioned using both Auto and Level3 ScanLevel, double-check if these levels are sufficient for the depth of classification you require. Sometimes, a higher scan level might be necessary for more detailed classifications.

Review the scan logs and reports for any errors or warnings that might indicate why data classifications are not being generated. These logs can provide insights into what might be going wrong.

I hope this information helps. Please do let us know if you have any further queries.

Kindly consider upvoting the comment if the information provided is helpful. This can assist other community members in resolving similar issues.
Jarrod Henningsen 20 Reputation points

2025-02-27T20:38:40.57+00:00

This is not helpful and it's actually a bit insulting. It's just the response from an LLM which has been copied and pasted.

What I want to know is why the default scan rule set (AdlsGen2 (System)) is not creating a classification even though Schema Classifications (All Full Names, Credit Card Number, Email Address, IP Address) have been found and should be triggering a classification since those rules are also in the scan rule set.
Anonymous

2025-02-28T20:41:27.0466667+00:00

@Jarrod Henningsen

I apologize for inconvenience,

The default scan rule set should include both schema and data classification rules. However, there might be a configuration issue where the data classification rules are not being applied correctly because It's possible that the specific data classification rules for items like full names, credit card numbers, email addresses, and IP addresses are not being triggered due to a mismatch in the rule definitions or patterns and some other factors as well

could you please provide screenshots of Rule set configuration and classification rules, if possible. We will resolve your issue shortly

please refer ;https://learn.microsoft.com/en-us/purview/create-a-scan-rule-set

Thank you
Jarrod Henningsen 20 Reputation points

2025-02-28T22:23:29.9+00:00

removed because I meant to post a comment instead of an answer
Jarrod Henningsen 20 Reputation points

2025-03-01T16:45:23.34+00:00

Thank you for your response phemanth. As shown in the picture below the Schema Classifications occur automatically during the scan, but I'm only able to assign Classification values manually through the Edit menu. The scan below was run using the default AzureStorage(System) scan rule set, but the result was the same. I also tried running a scan with the Custom Classification rule shown at the bottom, but it did not generate either classification.

Is it possible to have scans automatically populate value in the Classification section?

Answer accepted by question author

1 additional answer

Your answer

Anonymous

2025-02-27T20:09:11.9366667+00:00

@Jarrod Henningsen

Welcome to the Microsoft Q&A and thank you for posting your questions here.

When using Azure Purview to scan data in an ADLS Gen2 storage account, it's important to understand the distinction between schema classifications and data classifications. Schema classifications identify the structure of the data (like the types of fields in a CSV), while data classifications focus on the actual content of the data.

Here are troubleshooting steps; please check and confirm us whether the issue gets resolved:

Ensure that the scan rule set you are using (AdlsGen2 (System)) is configured to detect and classify the specific types of data you are expecting. Sometimes, custom rule sets might be needed.

Verify that the classification rules for data classifications are correctly defined and enabled. Schema classifications might be working because they are predefined, but data classifications might need additional configuration.

Check if the data sensitivity labels are properly applied and recognized by the system. Sometimes, the absence of these labels can prevent data classifications from being generated.

Ensure that the scanning service has the necessary permissions to access and classify the data within the ADLS Gen2 storage account.

Although you mentioned using both Auto and Level3 ScanLevel, double-check if these levels are sufficient for the depth of classification you require. Sometimes, a higher scan level might be necessary for more detailed classifications.

Review the scan logs and reports for any errors or warnings that might indicate why data classifications are not being generated. These logs can provide insights into what might be going wrong.

I hope this information helps. Please do let us know if you have any further queries.

Kindly consider upvoting the comment if the information provided is helpful. This can assist other community members in resolving similar issues.
Jarrod Henningsen 20 Reputation points

2025-02-27T20:38:40.57+00:00

This is not helpful and it's actually a bit insulting. It's just the response from an LLM which has been copied and pasted.

What I want to know is why the default scan rule set (AdlsGen2 (System)) is not creating a classification even though Schema Classifications (All Full Names, Credit Card Number, Email Address, IP Address) have been found and should be triggering a classification since those rules are also in the scan rule set.
Anonymous

2025-02-28T20:41:27.0466667+00:00

@Jarrod Henningsen

I apologize for inconvenience,

The default scan rule set should include both schema and data classification rules. However, there might be a configuration issue where the data classification rules are not being applied correctly because It's possible that the specific data classification rules for items like full names, credit card numbers, email addresses, and IP addresses are not being triggered due to a mismatch in the rule definitions or patterns and some other factors as well

could you please provide screenshots of Rule set configuration and classification rules, if possible. We will resolve your issue shortly

please refer ;https://learn.microsoft.com/en-us/purview/create-a-scan-rule-set

Thank you
Jarrod Henningsen 20 Reputation points

2025-02-28T22:23:29.9+00:00

removed because I meant to post a comment instead of an answer
Jarrod Henningsen 20 Reputation points

2025-03-01T16:45:23.34+00:00

Thank you for your response phemanth. As shown in the picture below the Schema Classifications occur automatically during the scan, but I'm only able to assign Classification values manually through the Edit menu. The scan below was run using the default AzureStorage(System) scan rule set, but the result was the same. I also tried running a scan with the Custom Classification rule shown at the bottom, but it did not generate either classification.

Is it possible to have scans automatically populate value in the Classification section?

Answer 1

Hello @Jarrod Henningsen ,

I understand your concern regarding the data classifications.

The below possible scenarios might be the reasons for the above behavior. Do let us know if not.

If the sample size is too small or not representative, the system may not confidently apply a classification. You need to ensure that your data contains sufficient and representative samples of the sensitive information you expect Purview to classify.
As per this MS Documentation, for system classification minimum match threshold value is set at 60% and cannot be changed. Cross-check whether your input csv data met this threshold or not.

Coming to your custom classification, please try the below possible workaround methods while creating it.

There might be some hard-coding mistake while giving the Regular expression and you can consider generating the automatic Regular expression by giving a sample csv file containing all required classification values. If needed, you can test the classifying rule with another set of csv file containing sample test cases for this classification.
You can try the Dictionary classification which also needs a csv file with single column of containing required values.

I hope this information helps.

If the answer is helpful, please click Accept Answer and kindly upvote it. If you have any further questions about this answer, please click Comment.

Anonymous

2025-03-05T07:06:31.3566667+00:00

Hi @Jarrod Henningsen , just checking in to see if any of the provided suggestions helped. And, if you have any further queries do let us know. If the above solution helps, please do not forget to click "Accept the answer” and Yes, this can be beneficial to other community members.

If you have any other questions or still running into more issues, let me know in the "comments" and I would be happy to help you.
Deleted

This comment has been deleted due to a violation of our Code of Conduct. The comment was manually reported or identified through automated detection before action was taken. Please refer to our Code of Conduct for more information.

Answer 2

Jarrod Henningsen 20

Thanks Rakesh. I will try testing with different custom classifications rules and datasets

Share via

Purview Scans are creating Schema Classifications but not Data Classifications

1 additional answer

Your answer