Unable to create hierarchy of entities using REST API upload

Ramkumar Ranganathan 6

We are trying to create a Hierarchy of entities of our on premise data flows.

We have successfully created entities such as Hive Server, Hive Database and Hive Table using POST request - {Endpoint}/catalog/api/atlas/v2/entity. We have also established a relationship between them using POST request - {Endpoint}/catalog/api/atlas/v2/relationship.

However, we are unable to see linkage under Hierarchy in the Overview tab of an entity in Azure Purview Studio.

Could you please suggest on how to create linkage between entities so that it is visible in Azure Purview Studio.

1 answer

KranthiPakala-MSFT 46,512 Reputation points Microsoft Employee

2021-10-11T21:34:37.017+00:00
Hi @amkumarranganathan-0960,

Welcome to Microsoft Q&A forum and thanks for posting your query.

The hierarchy shortcut links are based on the Qualified Name.

Example: For SQL server, you need to follow below pattern:

mssql://server/database/schema/table#column

If you have followed the above convention for your Hive entities, and have entities that represent the entire path along that qualified name, then you should see the hierarchy even when you create it via the API.

Hope this info helps. Do let us know us know if you have further query.

----------

Please don't forget to click on and upvote button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how

Want a reminder to come back and check responses? Here is how to subscribe to a notification

If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators
Please sign in to rate this answer.

2 people found this answer helpful.
Ramkumar Ranganathan 6 Reputation points

2021-10-12T10:41:11.723+00:00

Hi @KranthiPakala-MSFT ,

Thanks for your inputs.
Based on your suggestion we have successfully created a hierarchy of entities for types Oracle and Azure SQL.

However, we are unable to do the same for Hive. For Hive, we followed the pattern 'hive://server/database/table/columns'

On analyzing all the type definitions from our purview account, we notice keys "purviewEntityExtDef" and "qualifiedNamePattern" exists for entities of types Oracle, Azure SQL etc. that suggest a pattern for qualifiedName. But for type 'hive', the above mentioned keys and pattern does not exist.

We also tried to create a custom type definition using the keys "purviewEntityExtDef" and "qualifiedNamePattern" but were unable to form a hierarchy of custom entities.

Please help with forward path to create hierarchy of entities for Hive and Custom objects.

Best Regards,
Ramkumar

KranthiPakala-MSFT 46,512 Reputation points Microsoft Employee

2021-10-15T05:44:30.943+00:00

HI @Ramkumar Ranganathan ,

Glad to the above response was helpful.
On the follow-up query regarding forming hierarchy of custom entities, I will reach out to internal team to get additional info and will get back to you as soon as we have an update.

Thank you.

KranthiPakala-MSFT 46,512 Reputation points Microsoft Employee

2021-10-15T21:32:25.173+00:00

HI @Ramkumar Ranganathan ,

As per the discussion with internal team, it looks like Apache Atlas defines the qualified name in this way:

Apache Atlas – Hive Atlas Bridge

hive_db.qualifiedName - <dbName>@<clusterName>

hive_table.qualifiedName - <dbName>.<tableName>@<clusterName>

hive_column.qualifiedName - <dbName>.<tableName>.<columnName>@<clusterName>

It might be worth experimenting with this qualified named pattern for the Hive assets.

Also have confirmed with team that Custom entity types do NOT support this Hierarchy feature.

Thank you

Ramkumar Ranganathan 6 Reputation points

2021-10-18T07:41:59.38+00:00

Hi @KranthiPakala-MSFT

Thanks for the update.

We created the qualified names for Hive Assets as mentioned above. However, we were unable to see linkage under Hierarchy in the Overview tab.

Please let me know if you come across a solution for presenting Hive Assets in a hierarchical manner.

Best Regards,
Ramkumar

KranthiPakala-MSFT 46,512 Reputation points Microsoft Employee

2021-10-21T17:20:21.557+00:00

Hi @Ramkumar Ranganathan ,

Thanks for getting back, I'm reaching out to internal team to get additional assistance on this requirement. Will let you know once I have more details.

Thank you

KranthiPakala-MSFT 46,512 Reputation points Microsoft Employee

2021-10-22T23:38:23.257+00:00

Hi @Ramkumar Ranganathan ,

Could you please try with the below as the qualified name pattern for the Hive source and let us know how it goes on your end.

Entity type qualifiedName pattern Hive_db {dbName}@{clusterName} Hive_table {dbName}.{tableName}@{clusterName} Hive_storagedesc {dbName}.{tableName}@{clusterName}_storage Hive_column {dbName}.{tableName}.{columnName}@{clusterName} Hive_server {clusterName} Hive_view {dbName}.{viewName}@{clusterName} Hive_foreign_key {dbName}.{tableName}.{fkName}@{clusterName} Hive_unique_constraint {dbName}.{tableName}.{consName}@{clusterName} Hive_view_query {qualifiedName of view}:query

We look forward to your confirmation.

Thank you

KranthiPakala-MSFT 46,512 Reputation points Microsoft Employee

2021-10-25T18:10:17.477+00:00

Hi @Ramkumar Ranganathan ,

Just checking in to see if the above information was helpful. If it answers your query, please do click “Accept Answer” and/or Up-Vote, as it might be beneficial to other community members reading this thread. And, if you have any further query do let us know.

Ramkumar Ranganathan 6 Reputation points

2021-10-26T14:23:09.8+00:00

Hi @KranthiPakala-MSFT ,

Thanks for your support and inputs.

We created entities of type Hive_db, Hive_table and Hive_server using the above mentioned qualifiedName pattern. However, we are unable to see the entities in a hierarchical manner for Hive as it is visible for AzureSQL Server or Oracle Server

Secondly, would also like to mention that entities of Hive Server in "By Source Type" tab under "Data Catalog" are greyed out unlike AzureSQL Server or Oracle Server. Could this imply that hierarchies are not supported for type Hive?

Best Regards,
Ramkumar

KranthiPakala-MSFT 46,512 Reputation points Microsoft Employee

2021-10-29T05:14:26.787+00:00

Hi @Ramkumar Ranganathan ,

Thanks for getting back. I have reached out to product team to confirm on your following query. Will keep you posted as soon as I hear back from the team.

Thank you for your patience.

KranthiPakala-MSFT 46,512 Reputation points Microsoft Employee

2021-10-29T17:17:13.243+00:00

Hi @Ramkumar Ranganathan ,

Could you please confirm how are you scanning your Hive sources? Is it by Hive metastore data source?

Ramkumar Ranganathan 6 Reputation points

2021-11-01T07:19:14.69+00:00

Hi @KranthiPakala-MSFT ,

We are not using the scan functionality for Hive sources. We are creating entities using REST API.

Best Regards,
Ramkumar

KranthiPakala-MSFT 46,512 Reputation points Microsoft Employee

2021-11-04T18:18:53.213+00:00

Hi @Ramkumar Ranganathan ,

Thanks for your response. For a deeper investigation on this issue, if you have a support plan, could you please file a support ticket and do share the support ticket number with us here? In case if you don't have a support plan, please let us know here so that we can work with you offline.

Thank you

KranthiPakala-MSFT 46,512 Reputation points Microsoft Employee

2021-11-10T16:34:44.487+00:00

Hi @Ramkumar Ranganathan ,

Following up to see if you have got a chance to file a support ticket for deeper investigation. In case if you don't have a support plan please let me know here. In case if you have already filed a SR please do share the SR# with us here so that we can track it internally.

Thank you

Ramkumar Ranganathan 6 Reputation points

2021-11-11T06:27:27.383+00:00

Hi @KranthiPakala-MSFT ,

Sorry for the delay in response.

We do not have a support plan. Please suggest on how to proceed.

Best Regards,
Ramkumar

Behroz Ahmad Ali 0 Reputation points

2023-10-11T13:13:22.0766667+00:00

@KranthiPakala-MSFT Is there any update on this? So far none of the suggested answers have worked.

Guillaume Lepert 0 Reputation points

2024-06-25T08:29:24.5266667+00:00

Is there a reference detailing the Qualified Name requirements for every supported datasource? For example for PostgreSQL is should be of the form

postgres://servers/my-server:5432/dbs/postgres/schemas/public/tables/test_table/columns/column1

which is completely different. We also need to know the correct prefix for every source kind.
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

Share via

Unable to create hierarchy of entities using REST API upload

1 answer

Your answer