Purview Duplicates assets when scanning Power BI

Question

Purview Duplicates assets when scanning Power BI

Arnoldo Thiago Monteiro Lima 20

In Purview, I am confronted with duplicate data assets.

On one side the scan is running on top of Azure SQL Server. This scans a number of different assets between schemas, tables, views and so on.

On the other side the scan is performed on the Power BI Service.

When scanning Power BI, the scan ends up creating some table-type assets within Purview as the data source consumed by Power BI. However, most of the reports we have in Power BI consume data from Views.

This causes Purview to create duplicate assets: A table and a View with the same name and qualifiedName, changing only that the table created does not have any shema information. The problem is that because of this the data lineage is broken inside Purview, since it is not possible to create a custom data lineage for Power BI data sets.

Both assets have exactly the same qualifiedName:

Table: mssql://SQLServer.windows.net/DB/Schema/View
View: mssql://SQLServer.windows.net/DB/Schema/View

Is it possible to unify these two data assets through resource set or asset grouping?

Or is it possible to change the type of assets that Power BI generates when scanned?

Thanks in advance!

Bhargava-MSFT 31,261 Reputation points Microsoft Employee Moderator

2023-05-15T18:25:37+00:00
Hello Arnoldo Thiago Monteiro Lima,

Welcome to the MS Q&A platform.

My understanding is that this is expected behavior for Purview to detect two types of assets resource sets, as they are different types of assets with different properties.

However, it is possible to create a resource set for the duplicate assets. You can define a custom pattern rule that matches the qualified name of the assets and groups them together. For example, you can define a pattern rule that matches the qualified name of the assets and groups them together based on the schema name.

Please see the below to define a pattern rule for the duplicate assets:

Go to the "Resource set pattern rules" page in the Purview management center.

Create a new pattern rule.

In the "Pattern" field, enter a regular expression that matches the qualified name of the assets. For example, you could use a pattern like mssql://SQLServer\.windows\.net/DB/(.*)/View.

In the "Grouping" field, enter a grouping expression that extracts the schema name from the qualified name. For example, you could use a grouping expression like $1.

Click "Save" to create the pattern rule.

Reference document: https://learn.microsoft.com/en-us/azure/purview/how-to-resource-set-pattern-rules

https://github.com/MicrosoftDocs/azure-docs/blob/main/articles/purview/concept-resource-sets.md

Once you have defined the pattern rule, Purview will group the duplicate assets together into a single resource set based on the schema name.

I hope this helps. Please let me know if you have any further questions.
Arnoldo Thiago Monteiro Lima 20 Reputation points

2023-05-15T19:46:17.42+00:00

Hi BhargavaGunnam-MSFT,

Unfortunately resource sets only work for Accounts type: Azure Data Lake Storage Gen2, Azure Blob Storage, Azure Files and Amazon S3.

Is there any way to set a pattern rule for SQL Server?

Best Regards
Bhargava-MSFT 31,261 Reputation points Microsoft Employee Moderator

2023-05-15T20:38:16.63+00:00

Hi Arnoldo Thiago Monteiro Lima,

Thank you for the reply. Sorry, you are correct. The set pattern rule is not available for SQL servers yet. Let me check with my internal team and get back to you with more details.

Accepted answer

0 additional answers

Your answer

Bhargava-MSFT 31,261 Reputation points Microsoft Employee Moderator

2023-05-15T18:25:37+00:00

Hello Arnoldo Thiago Monteiro Lima,

Welcome to the MS Q&A platform.

My understanding is that this is expected behavior for Purview to detect two types of assets resource sets, as they are different types of assets with different properties.

However, it is possible to create a resource set for the duplicate assets. You can define a custom pattern rule that matches the qualified name of the assets and groups them together. For example, you can define a pattern rule that matches the qualified name of the assets and groups them together based on the schema name.

Please see the below to define a pattern rule for the duplicate assets:

Go to the "Resource set pattern rules" page in the Purview management center.

Create a new pattern rule.

In the "Pattern" field, enter a regular expression that matches the qualified name of the assets. For example, you could use a pattern like mssql://SQLServer\.windows\.net/DB/(.*)/View.

In the "Grouping" field, enter a grouping expression that extracts the schema name from the qualified name. For example, you could use a grouping expression like $1.

Click "Save" to create the pattern rule.

Reference document: https://learn.microsoft.com/en-us/azure/purview/how-to-resource-set-pattern-rules

https://github.com/MicrosoftDocs/azure-docs/blob/main/articles/purview/concept-resource-sets.md

Once you have defined the pattern rule, Purview will group the duplicate assets together into a single resource set based on the schema name.

I hope this helps. Please let me know if you have any further questions.
Arnoldo Thiago Monteiro Lima 20 Reputation points

2023-05-15T19:46:17.42+00:00

Hi BhargavaGunnam-MSFT,

Unfortunately resource sets only work for Accounts type: Azure Data Lake Storage Gen2, Azure Blob Storage, Azure Files and Amazon S3.

Is there any way to set a pattern rule for SQL Server?

Best Regards
Bhargava-MSFT 31,261 Reputation points Microsoft Employee Moderator

2023-05-15T20:38:16.63+00:00

Hi Arnoldo Thiago Monteiro Lima,

Thank you for the reply. Sorry, you are correct. The set pattern rule is not available for SQL servers yet. Let me check with my internal team and get back to you with more details.

Answer 1

Bhargava-MSFT 31,261 Microsoft Employee Moderator

Hi Arnoldo Thiago Monteiro Lima,

PG confirmed that this is a known issue from the PBI scan.

The workaround is to manually link via lineage or add a description/custom attribute of the duplicate assets.

I hope this helps. Please let me know if you have any further questions.

If this answers your question, please consider accepting the answer by hitting the Accept answer and up-vote as it helps the community look for answers to similar questions.

Share via

Purview Duplicates assets when scanning Power BI

0 additional answers

Your answer