Extracting the metadata from Purview in Json file using Purview CLI

Question

Extracting the metadata from Purview in Json file using Purview CLI

Maria Diana 45

I have successfully scanned few data sources in Purview UI. I would like to extract the metadata from purview in Json format so I can store the Json files in Jupyter 1.

So there are two things here, First one will be extracting the metadata from Purview. How can I do this. Is there any clear documentation somewhere? I have gone through some documentation and found that we can use purview cli to do it But I am not able access purview commands (pv commands) after installing purview cli through powershell. Powershell is not recognizing the pv commands (error “The term 'pv' is not recognized as the name of a cmdlet, function, script file, or operable program. Check the spelling of the name, or if a path was included, verify that the path is correct and try again”).

Can you please help me to understand how can I extract the metadata using purview cli in json format so that it can be moved to Jupyter 1?

KranthiPakala-MSFT 45,887 Reputation points Microsoft Employee

2023-10-26T19:47:57.75+00:00

@Maria Diana Welcome to Microsoft Q&A forum and thanks for reaching out here.

To better assist on this issue, could you please help share the commands you have used which resulted in above error message?

Thanks
Maria Diana 45 Reputation points

2023-10-27T10:35:58.68+00:00

First, I used the command <pip install purviewcli> to install purview cli and it has successfully been installed
Next, to access the purview commands, I have used the command <pv> and it's not able to recognize this command.
I have also tried to connect to the purview account using the command <az login> and <az-connect> and again its the same error as purview commands.

I have used the terminals - powershell, cmd and gitbash and the response is the same.
KranthiPakala-MSFT 45,887 Reputation points Microsoft Employee

2023-11-02T18:01:04.33+00:00

@Maria Diana Checking to see if you have got a chance to try the above instructions? If so, kindly let us know how it goes.

Thank you
Maria Diana 45 Reputation points

2023-11-07T12:01:45.73+00:00

Hello,

Thank you for the above. I have created the VENV environment and downloaded Azure CLI which helped the pv commands and az login command to work.

pv entity read --guid 'abc-123' – This command has helped to get the metadata at an Asset level.

Is there a command to pull the metadata at a collection level? The metadata for all the assets at once for a single collection.

Many Thanks.
Maria Diana
KranthiPakala-MSFT 45,887 Reputation points Microsoft Employee

2023-11-08T20:02:27.58+00:00

@Maria Diana Glad to know the provided info was helpful.

For collection level, have you got a chance to explore Collection related commands? If not, please give a try and let us know how it goes: https://github.com/tayganr/purviewcli/blob/master/doc/commands/account/main.md

Thank you
Maria Diana 45 Reputation points

2023-11-09T09:31:20.38+00:00

Hello,

Thank you for the response. I have explored the Collection related commands. The command pv account getCollection - only gives us the content on Collections and not the Assets inside the Collection. I would like to pull the metadata for all the assets at once for a single collection. Because, if there are 700 assets in a collection, then it becomes difficult to pull the metadata in json format for all the assets using the guid and using the command - pv entity read --guid 'abc-123'.

Is there any way I can pull the metadata at a collection level. With this I mean all the assets' metadata inside a collection.
KranthiPakala-MSFT 45,887 Reputation points Microsoft Employee

2023-11-13T21:24:45.43+00:00

@Maria Diana As per my conversation with product team, if the objective is to “get all assets within a single collection”, this is similar to what you are able to do via the GUI, by browsing to assets by collection (Data Catalog > Browse > By collection). There are ways to replicate this GUI behavior programmatically, but it is important to note, this won’t technically be “everything” that sits within a collection, as it will only retrieve the top-level entities, it will not retrieve the entities that hang off the results (e.g. schemas, columns, etc).

For example, when clicking on the “Marketing” link in the screenshot above, the portal will fire off a query to the search API, passing a filter for the specific collection.

You can see in the screenshot below; this result can be replicated in purviewcli using pv search query with an appropriate filter payload.

If you just need the top-level entities, this approach might be sufficed. If you need the children (e.g. schemas, columns, etc), then you will need to write code to programmatically loop through and get all the dependencies.

Hope this info helps.

KranthiPakala-MSFT 45,887 Reputation points Microsoft Employee

2023-10-26T19:47:57.75+00:00

@Maria Diana Welcome to Microsoft Q&A forum and thanks for reaching out here.

To better assist on this issue, could you please help share the commands you have used which resulted in above error message?

Thanks
Maria Diana 45 Reputation points

2023-10-27T10:35:58.68+00:00

First, I used the command <pip install purviewcli> to install purview cli and it has successfully been installed
Next, to access the purview commands, I have used the command <pv> and it's not able to recognize this command.
I have also tried to connect to the purview account using the command <az login> and <az-connect> and again its the same error as purview commands.

I have used the terminals - powershell, cmd and gitbash and the response is the same.
KranthiPakala-MSFT 45,887 Reputation points Microsoft Employee

2023-11-02T18:01:04.33+00:00

@Maria Diana Checking to see if you have got a chance to try the above instructions? If so, kindly let us know how it goes.

Thank you
Maria Diana 45 Reputation points

2023-11-07T12:01:45.73+00:00

Hello,

Thank you for the above. I have created the VENV environment and downloaded Azure CLI which helped the pv commands and az login command to work.

pv entity read --guid 'abc-123' – This command has helped to get the metadata at an Asset level.

Is there a command to pull the metadata at a collection level? The metadata for all the assets at once for a single collection.

Many Thanks.
Maria Diana
KranthiPakala-MSFT 45,887 Reputation points Microsoft Employee

2023-11-08T20:02:27.58+00:00

@Maria Diana Glad to know the provided info was helpful.

For collection level, have you got a chance to explore Collection related commands? If not, please give a try and let us know how it goes: https://github.com/tayganr/purviewcli/blob/master/doc/commands/account/main.md

Thank you
Maria Diana 45 Reputation points

2023-11-09T09:31:20.38+00:00

Hello,

Thank you for the response. I have explored the Collection related commands. The command pv account getCollection - only gives us the content on Collections and not the Assets inside the Collection. I would like to pull the metadata for all the assets at once for a single collection. Because, if there are 700 assets in a collection, then it becomes difficult to pull the metadata in json format for all the assets using the guid and using the command - pv entity read --guid 'abc-123'.

Is there any way I can pull the metadata at a collection level. With this I mean all the assets' metadata inside a collection.
KranthiPakala-MSFT 45,887 Reputation points Microsoft Employee

2023-11-13T21:24:45.43+00:00

@Maria Diana As per my conversation with product team, if the objective is to “get all assets within a single collection”, this is similar to what you are able to do via the GUI, by browsing to assets by collection (Data Catalog > Browse > By collection). There are ways to replicate this GUI behavior programmatically, but it is important to note, this won’t technically be “everything” that sits within a collection, as it will only retrieve the top-level entities, it will not retrieve the entities that hang off the results (e.g. schemas, columns, etc).

For example, when clicking on the “Marketing” link in the screenshot above, the portal will fire off a query to the search API, passing a filter for the specific collection.

You can see in the screenshot below; this result can be replicated in purviewcli using pv search query with an appropriate filter payload.

If you just need the top-level entities, this approach might be sufficed. If you need the children (e.g. schemas, columns, etc), then you will need to write code to programmatically loop through and get all the dependencies.

Hope this info helps.

Answer 1

@Maria Diana Thanks for the additional details.

All of the tayganr/purviewcli: Microsoft Purview CLI (github.com) are NOT powershell, it is its own shell command.

Video: Could you please try Visual Studio Code instead of Powershell as demonstrated in this video: CLI wrapper to Microsoft Purview APIs (purviewcli)

You should just call pv entity read --guid 'abc-123' and it will pull the content down in json.

If you would like to explore Powershell cmdlet, it's a different pacakge: Azure/Azure-Purview-API-PowerShell (github.com)

Hope this info helps. Let us know how it goes.

Thank you

Please don’t forget to Accept Answer and Yes for "was this answer helpful" wherever the information provided helps you, this can be beneficial to other community members.

Answer 2

Ravineesh 20

@Maria Diana , Have you tried running the command pv search query ? I think you can get the data for collection level by using pv search query

Also have a look at this https://github.com/mdrakiburrahman/purview-collection-aggregator/blob/main/explore.ipynb it might be useful

Extracting the metadata from Purview in Json file using Purview CLI

1 additional answer

Activity