Microsoft Purview - Data Profiling

Question

Hi Team,

I would like to setup some rules in Microsoft Purview, to scan my Azure SQL tables and give me a data score on my data quality like missing data, Nulls, Inconsistent data etc based on some custom rules.

Does the new version of the Purview support this? as I do not see all options mentioned in the documentation are not seen in the Purview account when I login in.

What are the other options to achieve this?

Thanks,

Nalini.

Answer 1

Hi Nalini Bhavaraju

Thank you for reaching out microsoft Q&A!

The Microsoft Purview Unified Catalog experience does support configuring out-of-the-box and custom data quality rules to evaluate Azure SQL data, including checks for empty or null values, duplicates, format conformity, and other validation logic. These rules generate data quality scores that appear within the Health management > Data quality experience after profiling and scans are completed. Some rule types may vary by source, and certain rules (for example freshness) may not apply to Azure SQL.

If you are not seeing the Data Quality options in your portal, it is usually related to prerequisites such as Data Governance capacity being enabled, correct role assignment (for example Data Quality Steward), completed scans in the Data Map, and assets being onboarded into a governance domain or data product. Also ensure the Azure SQL source is configured with the required permissions for the Purview managed identity.

You can review the official documentation here for detailed setup and supported capabilities: https://learn.microsoft.com/en-us/purview/unified-catalog-data-quality-rules https://learn.microsoft.com/en-us/purview/unified-catalog-data-quality-supported-sources-connection https://learn.microsoft.com/en-us/purview/whats-new

If possible, please share a screenshot of what you currently see in the Purview portal along with the region of your Purview account, and we can help verify whether this is a configuration or feature-availability issue.

Answer 2

Yes, Microsoft Purview supports setting up data quality rules to scan your Azure SQL tables and assess data quality, including aspects like missing data, nulls, and inconsistent data. You can configure various out-of-the-box data quality rules such as:

1. Freshness - Checks if the asset is updated within the expected time.
2. Empty/blank fields - Looks for null or empty values in specified columns.
3. Unique values - Ensures that all values in a specified column are unique.
4. Data type match - Verifies that values in a column match their expected data types.
5. Duplicate rows - Checks for duplicate rows across specified columns.
6. String format match - Validates that values in a column match a specific format.
7. Table lookup - Confirms that values in one table exist in another specified table.
8. Custom rules - Allows you to create specific rules using visual expression builders or custom expressions.

To set this up, you would typically follow the data quality life cycle, which includes registering your data source, scanning it, configuring data profiling, and then applying the data quality rules to your data asset. If you do not see all options available in your Purview account, ensure that you have the necessary permissions and that your account is set up correctly to access all features.

If you encounter limitations, consider reaching out to Microsoft support or consulting the documentation for troubleshooting steps or updates regarding your Purview account.

---

References:

• Overview of data quality in Microsoft Purview Unified Catalog
• Available data quality rules