Đọc bằng tiếng Anh

Chia sẻ qua

Data quality for Snowflake data

  • Bài viết

When scanning Snowflake data sources, Microsoft Purview supports extracting technical metadata, including:

  • Server, databases, schemas, and tables, including the columns; views including the columns, foreign keys, and unique constraints.
  • Stored procedures, including the parameter dataset and result set.
  • Functions, including the parameter dataset.
  • Pipes, stages, and streams, including the columns.
  • Tasks and sequences.

Configure Data Map scan to catalog Snowflake data in Microsoft Purview

Register a Snowflake source

To register a new Snowflake source in Unified Catalog, follow these steps:

  • Sign into the Microsoft Purview portal.
  • Select the Data Map solution card. If the Data Map solution card isn't displayed, select View all solutions and then select Data Map from the Core section.
  • Select Register.
  • On Register sources, select Snowflake.

Set up a Data Map scan

  • Select the registered Snowflake source.
  • Select + New scan.
  • Provide the below details:
    • Name: The name of the scan
    • Connect via integration runtime: Select the Azure autoresolved integration runtime, Managed VNet IR, or SHIR according to your scenario.
    • Host for connection: Choose the endpoint used to establish connection to Snowflake during scan. You can choose from the server URL, or the other hosts that you configured in data source.
    • Credential: Select the credential to connect to your data source. Make sure to:
      • Select Basic Authentication while creating a credential.
      • Provide the user name used to connect to Snowflake in the User name input field.
      • Store the user password used to connect to Snowflake in the secret key.
    • Warehouse: Specify the name of the warehouse instance used to empower scan in capital case. The default role assigned to the user specified in the credential must have USAGE rights on this warehouse.
    • Databases: Specify one or more database instance names to import in capital case. Separate the names in the list with a semi-colon (;). For example, DB1;DB2. The default role assigned to the user specified in the credential must have adequate rights on the database objects.
    • Schema: List subset of schemas to import expressed as a semicolon separated list.
  • Select Test connection to validate the settings (available when using Azure Integration Runtime).
  • Select Continue.
  • Select a scan rule set for classification. You can choose between the system default, existing custom rule sets, or create a new rule set inline.
  • Review your scan and select Save and Run.

Once scanned, the data asset in Snowflake will be available on the Unified Catalog search. For more details about how to connect and manage Snowflake in Microsoft Purview, follow this document.

Quan trọng

When the object is deleted from the data source, the subsequent scan won't automatically remove the corresponding asset in Microsoft Purview.

Set up connection to Snowflake data source for data quality scan

At this point, the scanned asset is ready for cataloging and governance. Associate the scanned asset to the data product in a Governance Domain Sele. At the Data Quality Tab, add a new Azure SQL Database Connection: Get the Database Name entered manually.

  1. Select Data quality > Governance Domain > Manage tab to create connection.

    Screenshot that shows connection page navigation.

  2. Configure connection in the connection page.

    • Add connection name and description
    • select source type Snowflake
    • add Server name, Warehouse name, Database name, Schema name, and Table name
    • select authentication method - Basic authentication
    • add User name
    • add Credential
      • Add Azure subscription
      • Key vault connection
      • secret name
      • secret version

  3. Test connection to make sure the configured connection is successfully established.

    Screenshot that shows how to set up snowflake connection.

    Screenshot that shows how to configure snowflake connection token.

Quan trọng

  • Data quality stewards need read only access to Snowflake to set up the data quality connection.
  • vNet isn't supported for the Snowflake data source yet.
  • The Snowflake connector doesn't accept https://. Remove https:// when you add the server name to configure data source connection.

Profiling and data quality scanning for data in Snowflake

After completed connection setup successfully, you can profile, create and apply rules, and run DQ scan of your data in Snowflake. Follow the step-by-step guideline described in below documents:

Reference documents