Enabling Data Lineage for Google Cloud Storage and Google Big Query

Question

How to set up the Google Cloud Storage, and Google Big Query as data sources in Purview?

I have an established between the Azure Data Factory and Purview.

There are several pipelines where the data source is Google Cloud Storage or Google Big Query, the pipeline contains Copy Data and Data Flow activities, but when I check the data lineage status in Data Factory for Copy Data and Data Flow activities, I see the below error:

The lineage of this copy activity is not reported because it used the following unsupported connector: GoogleBigQuery.

The lineage of this copy activity is not reported because it used the following unsupported connector: GoogleCloudStorage.

I've registered the data sources in Purview as shown below

1. What are the limitations of Google Cloud Storage and Google Big Query in the context of Purview?
2. What are the limitations of the scan rule set that are defined in Purview for Google Cloud Storage and Google Big Query?

Answer 1

Hello Ravineesh,

Please see the below document for the limitations

• Currently, Microsoft Purview only supports scanning Google BigQuery datasets in US multi-regional location. If the specified dataset is in other location e.g. us-east1 or EU, you will observe scan completes but no assets shown up in Microsoft Purview.
• When object is deleted from the data source, currently the subsequent scan won't automatically remove the corresponding asset in Microsoft Purview.

https://learn.microsoft.com/en-us/purview/register-scan-google-bigquery-source#known-limitations

Scan: https://learn.microsoft.com/en-us/purview/register-scan-google-bigquery-source#scan

I hope this answers your question. Please let me know if you have any further questions.