Can we check Duplicates in Purview?

Question

Can we check Duplicates in Purview?

Maria Diana 45

Hello,

Can we check for duplicates in purview? For example; If the data in SQL Server database and the data in Azure Cosmos DB is same or partially same and I scan these both in purview, How can I find out that the data in both the data sources are same?

Many Thanks,
Maria Diana

Answer 1

BhargavaGunnam-MSFT 21,336 Microsoft Employee

Hello Maria Diana,

Welcome to the Microsoft Q&A forum.

Unfortunately, Microsoft Purview doesn't have an inbuilt feature to detect duplicate data between two registered data sources(Ex: SQL server and Azure cosmos DB)

However, there are some workarounds(as discussed in the below document) that you can use to identify duplicate data in your scenario. One approach is to use a data integration tool such as Azure Data Factory to extract data from both data sources and compare them using a data comparison tool. You can then use the results of the comparison to identify any duplicate data.

https://learn.microsoft.com/en-us/azure/architecture/databases/architecture/profisee-master-data-management-purview

https://learn.microsoft.com/en-us/azure/data-factory/how-to-data-flow-dedupe-nulls-snippets

I see a similar feedback item requested in the Purview feedback channel. I suggest upvote and comment on this feedback item. All the feedback shared in this forum is actively monitored and reviewed by the engineering team, who will prioritize them accordingly based on the number of requests.

https://feedback.azure.com/d365community/idea/1f4f3c29-da59-ec11-a819-0022484e8090

I hope this helps. Please let me know if you have any further questions.

BhargavaGunnam-MSFT 21,336 Reputation points Microsoft Employee

2023-11-08T17:39:09.8566667+00:00

Hello Maria Diana,

I am checking to see if you have any further questions.

If this answers your question, please consider accepting the answer by hitting the Accept answer and up-vote as it helps the community look for answers to similar questions.
Maria Diana 45 Reputation points

2023-11-09T10:56:23.46+00:00

Hello,

Just to confirm, Is there any way we could find the duplicates using with the metadata after the scan.

Many Thanks,
Maria Diana
BhargavaGunnam-MSFT 21,336 Reputation points Microsoft Employee

2023-11-09T18:20:17.2166667+00:00

Hi Maria Diana,

Unfortunately, even with the metadata after the scan, Microsoft Purview does not have an inbuilt feature to detect duplicate data between two registered data sources.

However, As I mentioned, One approach is to use a data integration tool such as Azure Data Factory to extract data from both data sources and compare them using a data comparison tool. You can then use the results of the comparison to identify any duplicate data.

For near-duplicate detection, Microsoft has a separate service in eDiscovery. When near duplicate detection is run, the system parses every document with text. Then, it compares every document against each other to determine whether their similarity is greater than the set threshold. If it is, the documents are grouped together.

The Near Duplicate Detection feature is designed to identify near-duplicate data(which may not be the same as exact duplicates).

You may still need to use a data integration tool or a third-party tool to identify exact duplicates in your data sources.

This is a premium feature and requires a separate license.

https://learn.microsoft.com/en-us/purview/ediscovery-near-duplicate-detection

https://learn.microsoft.com/en-us/purview/ediscovery-configure-search-and-analytics-settings

I hope this helps.

Can we check Duplicates in Purview?

0 additional answers

Activity