MS Purview Data Catalogue DCAT Schema Support and Ideal Public Frontend

Question

MS Purview Data Catalogue DCAT Schema Support and Ideal Public Frontend

Herbert Kimani 56

Does Purview support API consumption using DCAT to build a federated data catalogue and is there an example implementation of this and what was or would be the ideal frontend technology to build a public/online advanced/faceted search app with Data Analysis and future AI integration? e.g. Sharepoint site, PowerApps, a custom SPA application, etc?

PRADEEPCHEEKATLA 90,606 Reputation points

2023-11-28T08:24:44.0866667+00:00

@Herbert Kimani - Thanks for the question and using MS Q&A platform.

Unfortunately, Microsoft Purview doesn't support API consumption using DCAT to build a federated data catalogue.

Micorosft Purview has Federated Data Catalogue capability. With Purview, you can create a centralized data catalog that provides a unified view of all your data assets, regardless of where they are located. This includes data stored in Azure, on-premises, and in other cloud environments.

Purview achieves this through its support for a wide range of data connectors, which allow you to connect to and scan data sources across your entire organization. These connectors include native support for Azure services such as Azure Data Factory, Azure Blob Storage, and Azure Synapse Analytics, as well as support for a wide range of third-party data sources such as Amazon S3, Google Cloud Storage, and more.

In addition to its support for data connectors, Purview also provides a range of features that make it easy to discover, understand, and manage your data assets. These include data lineage tracking, data classification and labeling, and business glossaries, among others.

Overall, Purview's Federated Data Catalogue capability makes it easy to create a centralized view of all your data assets, regardless of where they are located. This can help you to gain a better understanding of your data, improve data governance, and make more informed business decisions.

This guide Design a collection structure for a Microsoft Purview federated catalog describes a recommended structure for collections in a Microsoft Purview federated catalog. This design can help your organization avoid data silos.

When you build a federated catalog by using Microsoft Purview, it's important to consider the structure of the collections. This diagram shows the recommended structure for resolving Contoso's data silo challenge:

Hope this helps. Do let us know if you any further queries.
Herbert Kimani 56 Reputation points

2023-11-28T08:48:02.5633333+00:00

This seems to be tailored for internal use only. What about a public frontend for open data? Also, are you saying that I cannot consume metadata from external APIs and cannot also specify a standard like DCAT in the catalogue?
PRADEEPCHEEKATLA 90,606 Reputation points

2023-11-28T09:16:46.65+00:00

@Herbert Kimani - As I said above, Microsoft Purview doesn't support API consumption using DCAT to build a federated data catalogue.

Let me double check with the product team on the your ask:

What about a public frontend for open data? Also, are you saying that I cannot consume data from external APIs and cannot also specify a standard like DCAT in the catalogue?

We are reaching out to the internal team to get more information related to your query and will get back to you as soon as we have an update.
Arindam Bandyopadhyay 16 Reputation points Microsoft Employee

2023-12-07T11:29:41.7266667+00:00
@Herbert Kimani what exactly do you mean by "What about a public frontend for open data? Also, are you saying that I cannot consume metadata from external APIs and cannot also specify a standard like DCAT in the catalogue?"

Are you looking to pull metadata from Purview catalog to build a new federated data catalog using another standardized framework such as DCAT ? Or are you looking to pull metadata from another framework in to Purview?

If yes, sure you can use the Purview search and entity bulk export APIs to achieve this, but you've need to do the transcoding between API schemas (Atlas to DCAT or any other which you're using) since Purview is built on Atlas and we currently do not support any other schema transcoding as it is. It can be custom built though.

In future, however, we will support certain other frameworks such as OCSF (on the security domain, no timeline can be promised or quoted as of now) since there will be greater push towards open security frameworks, similar to what Amazon AWS has been doing. If Purview doesn't support any particular schema, it means it has not been prioritized as a business decision and not anything else. In that case, as I mentioned, you'll need to do the schema transformations/transcoding yourself.

Hope that helps!
Herbert Kimani 56 Reputation points

2023-12-08T08:48:37.28+00:00

@Arindam Bandyopadhyay thank you for your detailed answer. Some background info, I am not dealing with internal operational / admin data; I am dealing with scientific data that needs to be catalogued and shared openly with the world, in a tool that allows data discovery, advance search, LLM, etc. A previous answer here told me that purview DC is not a public facing tool like ckan or dkan. So what I'm asking is whether purview can be used as Data Catalogue service and then one can build a frontend or saas app to interact with the DC service. The frontend would be a publicly available (open access / data) application and would have the functionality described by relying on API calls to the DC service. The DC service would use DCAT (the defacto std for such) to make sense of all the data it is fed. i.e. I would add data to the DC from various internal and partner APIs, excel uploads, etc and map it to the relevant DCAT fields. I hope this clarifies my question.

Your answer

Herbert Kimani 56 Reputation points

2023-11-28T08:48:02.5633333+00:00

This seems to be tailored for internal use only. What about a public frontend for open data? Also, are you saying that I cannot consume metadata from external APIs and cannot also specify a standard like DCAT in the catalogue?
PRADEEPCHEEKATLA 90,606 Reputation points

2023-11-28T09:16:46.65+00:00

@Herbert Kimani - As I said above, Microsoft Purview doesn't support API consumption using DCAT to build a federated data catalogue.

Let me double check with the product team on the your ask:

What about a public frontend for open data? Also, are you saying that I cannot consume data from external APIs and cannot also specify a standard like DCAT in the catalogue?

We are reaching out to the internal team to get more information related to your query and will get back to you as soon as we have an update.
Arindam Bandyopadhyay 16 Reputation points Microsoft Employee

2023-12-07T11:29:41.7266667+00:00

@Herbert Kimani what exactly do you mean by "What about a public frontend for open data? Also, are you saying that I cannot consume metadata from external APIs and cannot also specify a standard like DCAT in the catalogue?"

Are you looking to pull metadata from Purview catalog to build a new federated data catalog using another standardized framework such as DCAT ? Or are you looking to pull metadata from another framework in to Purview?

If yes, sure you can use the Purview search and entity bulk export APIs to achieve this, but you've need to do the transcoding between API schemas (Atlas to DCAT or any other which you're using) since Purview is built on Atlas and we currently do not support any other schema transcoding as it is. It can be custom built though.

In future, however, we will support certain other frameworks such as OCSF (on the security domain, no timeline can be promised or quoted as of now) since there will be greater push towards open security frameworks, similar to what Amazon AWS has been doing. If Purview doesn't support any particular schema, it means it has not been prioritized as a business decision and not anything else. In that case, as I mentioned, you'll need to do the schema transformations/transcoding yourself.

Hope that helps!
Herbert Kimani 56 Reputation points

2023-12-08T08:48:37.28+00:00

@Arindam Bandyopadhyay thank you for your detailed answer. Some background info, I am not dealing with internal operational / admin data; I am dealing with scientific data that needs to be catalogued and shared openly with the world, in a tool that allows data discovery, advance search, LLM, etc. A previous answer here told me that purview DC is not a public facing tool like ckan or dkan. So what I'm asking is whether purview can be used as Data Catalogue service and then one can build a frontend or saas app to interact with the DC service. The frontend would be a publicly available (open access / data) application and would have the functionality described by relying on API calls to the DC service. The DC service would use DCAT (the defacto std for such) to make sense of all the data it is fed. i.e. I would add data to the DC from various internal and partner APIs, excel uploads, etc and map it to the relevant DCAT fields. I hope this clarifies my question.

Share via

MS Purview Data Catalogue DCAT Schema Support and Ideal Public Frontend

Your answer