Microsoft Purview disaster recovery, and migration best practices
Article
This article provides guidance on backup and recovery strategy when your organization has Microsoft Purview unified governance solutions in production deployment. You can also use this general guideline to implement account migration. The scope of this article is to cover manual BCDR methods, where you could automate using APIs.
Azure data center outages are rare, but can last anywhere from a few minutes to hours. Data Center outages can cause disruption to environments that are being relied on for data governance. By following the steps detailed in this article, you can continue to govern your data in the event of a data center outage for the primary region of your Microsoft Purview account.
Business continuity and disaster recovery (BCDR) in a Microsoft Purview instance refers to the mechanisms, policies, and procedures that enable your business to protect data loss and continue operating in the face of disruption, particularly to its scanning, catalog, and insights tiers. This page explains how to configure a disaster recovery environment for Microsoft Purview.
Today, Microsoft Purview doesn't support automated BCDR. Until that support is added, you're responsible to take care of backup and restore activities. You can manually create a secondary Microsoft Purview account as a warm standby instance in another region.
The following steps summarize how you can achieve disaster recovery manually:
Once the primary Microsoft Purview account is created, create one or more secondary Microsoft Purview accounts in a separate region.
Important
Microsoft Purview currently supports a single Microsoft Purview instance per tenant. To create a second account for backup and disaster recovery, contact support
All activities performed on the primary Microsoft Purview account must be carried out on the secondary Microsoft Purview accounts as well. This includes:
Maintain Account information
Create and maintain custom scan rule sets, classifications, and classification rules
Register and scan sources
Create and maintain collections along with the association of sources with the collections
Create and maintain credentials used while scanning
Curate data assets
Create and maintain glossary terms
Specific steps to create and maintain a disaster recovery account are provided later in the article. Before you follow them, read through the limitations and considerations.
Limitations and considerations
As you create your manual BCDR plan, keep the following points in mind:
You'll be charged for primary and secondary Microsoft Purview instances.
The primary and secondary Microsoft Purview accounts can't be configured to the same Azure Data Factory, Azure Data Share and Synapse Analytics accounts, if applicable. As a result, the lineage from Azure Data Factory and Azure Data Share can't be seen in the secondary Microsoft Purview accounts. This limitation will be addressed when automated BCDR is supported.
The integration runtimes are specific to a Microsoft Purview account. So, scans need to run in primary and secondary Microsoft Purview accounts in-parallel, multiple self-hosted integration runtimes must be maintained. This limitation will also be addressed when automated BCDR is supported.
Parallel execution of scans from both primary and secondary Microsoft Purview accounts on the same source can affect the performance of the source. This can result in scan durations to vary across the Microsoft Purview accounts.
It isn't advisable to back up "scanned" assets' details. You should only back up the curated data such as mapping of classifications and glossaries on assets. The only case when you need to back up assets' details is when you have custom assets via custom typeDef.
The backed-up asset count should be fewer than 100,000 assets. The main driver is that you have to use the search query API to get the assets, which have limitation of 100,000 assets returned. However, if you're able to segment the search query to get smaller number of assets per API call, it's possible to back up more than 100,000 assets.
If you want to continuously "sync" assets between two accounts, there are other steps that won't be covered in detail in this article. You have to use Microsoft Purview's Event Hubs to subscribe and create entities to another account. However, Event Hubs only has Atlas information. Microsoft Purview has added other capabilities such as glossaries and contacts that won't be available via Event Hubs.
Maintain Account information by granting access for the admin and/or service principal to the account at root level
Collections
Create and maintain Collections along with the association of sources with the Collections. You can call List Collections API and then get specific details of each collection via Get Collection API
Create and maintain credentials used while scanning. There's no API to extract credentials, so this must be redone in the new account.
Self-hosted integration runtime (SHIR)
Get a list of SHIR and get updated keys from the new account then update the SHIRs. This must be done manually inside the SHIRs' hosts. These need to be running before you create scans.
ADF connections
Currently an ADF can be connected to one Microsoft Purview at a time. You must disconnect ADF from failed Microsoft Purview account and reconnect it to the new account later.
This will populate all assets with default typedef. There are several reasons to run the scans again vs. exporting the existing assets and importing to the new account:
There's a limit of 100,000 assets returned from the search query to export assets.
It's cumbersome to export assets with relationships.
When you rerun the scans, you'll get all relationships and assets details up to date.
Microsoft Purview comes out with new features regularly so you can benefit from other features when running new scans.
Running the scans is the most effective way to get all assets of data sources that Microsoft Purview is already supporting.
To identify all custom typedef, you can use the get all type definitions API. This will return each type. You need to identify the custom types in such format as "serviceType": "<custom_typedef>"
Custom assets
To export custom assets, you can search those custom assets and pass the proper custom typedef via the discovery API
Note
There is a 100,000 return limit per search result.
You might have to break the search query so that it won’t return more than 100,000 records.
There are several ways to scope down the search query to get a subset of assets:
Using Keyword: Pass the parent FQN such as Keyword: "<Parent String>/*"
Using Filter: Include assetType with the specific custom typedef in your search such as "assetType": "<custom_typedef>"
Here's an example of a search payload by customizing the keywords so that only assets in specific storage account (exampleaccount) are returned:
You need to migrate the term templates from typedef output as well.
When you re-create the custom entities, you might need to prepare the payload prior to sending to the API:
Note
The initial goal is to migrate all entities without any relationships or mappings. This will avoid potential errors.
All timestamp value must be null such as updateTime, updateTime, and lastModifiedTS.
The guid can't be regenerated exactly as before so you have to pass in a negative integer such as "-5000" to avoid error.
The content of relationshipAttributes shouldn't be a part of the payload to avoid errors since it's possible that the guids aren't the same or haven't been created yet. You have to turn relationshipAttributes into an empty array prior to submitting the payload.
meanings contains all glossary mappings, which will be updated in bulk after the entities are created.
Similarly, classifications needs to be an empty array as well when you submit the payload to create entities since you have to create classification mapping to bulk entities later using a different API.
Migrate relationships
To complete the asset migration, you must remap the relationships. There are three tasks:
Call the relationship API to get relationship information between entities by its guid
Prepare the relationship payload so that there's no hard reference to old guids in the old Microsoft Purview accounts. You need to update those guids to the new account's guids.
Before migrating terms, you need to migrate the term templates. This step should be already covered in the custom typedef migration.
Using the Microsoft Purview governance portal
The quickest way to migrate glossary terms is to export terms to a .csv file. You can do this using the Microsoft Purview governance portal.
Using Microsoft Purview API
To automate glossary migration, you first need to get the glossary guid (glossaryGuid) via List Glossaries API. The glossaryGuid is the top/root level glossary guid.
The below sample response will provide the guid to use for subsequent API calls:
"guid": "c018ddaf-7c21-4b37-a838-dae5f110c3d8"
Once you have the glossaryGuid, you can start to migrate the terms via two steps:
The prerequisite for this step is to have all classifications available in the new account from Migrate configuration items step.
You must call the discovery API to get the classification assignments to assets. This is applicable to all assets. If you've migrated the custom assets, the information about classification assignments is already available in classifications property. Another way to get classifications is to list classification per guid in the old account.
If you have extracted asset information from previous steps, the contact details are available from the discovery API.
To assign contacts to assets, you need a list of guids and identify all objectid of the contacts. You can automate this process by iterating through all assets and reassign contacts to all assets using the Create Or Update Entities API.
Administer an SQL Server database infrastructure for cloud, on-premises and hybrid relational databases using the Microsoft PaaS relational database offerings.