Can you create an empty catalog?

Question

Greetings,

Azure Purview is new to me, but data cataloging tools are not, so what I wanted to ask I don't know if functionally Purview can even do. Additionally, these requirements were provided to me by a non-technical person whom knowledge of the Purview tools strictly from reading documentation (no application of the concepts/functionality).

So effectively what they want the following activities as part of a PoC for Purview:

1. Create data catalog with custom fields (manually)
   • Add data set in catalog
2. Update catalog from data pipeline using API
   -TBD – but e.g. metadata like: last update date, latest data quality report link
3. Create “Glossary of Terms” which are specific to data definition. (manual process)

My biggest area of confusion is understanding two of the above requests (#1 and #2). Can #1 & #2 can even be done through the Azure Purview tool? That is the creation of an empty catalog, but with the required fields wanted, and then through the use of the API populate that empty catalog/data set with data from a API pipeline from another source.

The requirement even to someone whom is not familiar with the Purview tool seems convoluted and functionally something that just can not be done through the tool.

ANY HELP ON THIS AND GUIDING ME THROUGH THESE QUESITONS WOUDL BE EMMENSELY APPRECIATED!!!

Answer 1

Hello @Anonymous ,

Thanks for the question and using MS Q&A platform.

As per my understanding you would like to know if Azure purview has capability to achieve the steps mentioned in the above ask, if so, how? Please correct if I'm not accurate. Based on my understanding of the ask below are few pointers which can get you started with:

• Data Catalog: Purview's Data Catalog supports 100's of different "types". Every asset in your catalog can have one type and each type can have their own attributes that are unique to the data source. In this way, you can create custom types to support any metadata you want to capture. a) You can create a custom type which supports adding any custom field you want but doesn't work with built-in connectors. Azure Purview REST API Deep Dive - YouTube b) A Purview Service / Data Catalog is provisioned empty and then you can add custom types.
• Update catalog using API: a) Yes, this absolutely can be done: How to use REST APIs for Azure Purview Data Planes - Azure Purview | Microsoft Learn b) I'd strongly encourage you to use PyApacheAtlas: wjohnson/pyapacheatlas: A python package to help work with the apache atlas REST APIs (github.com) c) You can also take advantage of some or our Custom Connector Solutions to accelerate your custom pushing to Purview: https://github.com/microsoft/Purview-Custom-Connector-Solution-Accelerator d) There are also examples available using PyApacheAtlas to query databases and extract the metadata: pyapacheatlas/samples/metadataIngest at master · wjohnson/pyapacheatlas (github.com) and pyapacheatlas/samples/redshift at master · wjohnson/pyapacheatlas (github.com)
• Create "Glossary of Terms": a) Yes, this can be done manually in the UI : How to create, import, and export glossary terms - Azure Purview | Microsoft Learn b) You can also do this using REST API : Azure Purview -Catalog Data Plane - Glossary - REST API Operations

Hope this will help. Please let us know if any further queries.

------------------------------

• Please don't forget to click on or upvote button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
• Want a reminder to come back and check responses? Here is how to subscribe to a notification
• If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators