How can I scan views from the same Azure Synapse Analytics schema to multiple collections?

HR 0

I have multiple collections in Microsoft Purview. I have registered an Azure Synapse Analytics source to a parent collection. Depending on the ownership of the datasets I need to scan in views from the same schema to different collections.

When I scan a subset of views from 1 schema to a collection, views that have been added to a different collection using the same schema are removed. The views that are added to these 2 collections from the same schema are different. I nee to ensure the views are not removed from either collection.

Any help would be appreciated.

Harishga 3,425 Reputation points Microsoft Vendor

2024-04-22T12:01:43.7233333+00:00

@HR
Following up to see if the below answer was helpful. If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.

2 answers

Amira Bedhiafi 15,446 Reputation points

2024-04-18T10:13:53.23+00:00

In Microsoft Purview, asset mappings allow you to determine how different assets are assigned to collections based on specific criteria. You can set up asset mapping rules that differentiate the assets based on your required criteria (for example naming conventions, metadata).

https://learn.microsoft.com/en-us/purview/how-to-metamodel

You can configure scoped scans to only include specific views in each scan. This way, you can ensure that only the views relevant to a particular collection are scanned into that collection, without affecting the views in other collections.

Customize the settings of each scan to target different views. You can do this by specifying filters or SQL queries within the scan settings that limit the scope of the scan to only the views you want to include in each collection.
Please sign in to rate this answer.

0 comments No comments
Sign in to comment
Harishga 3,425 Reputation points Microsoft Vendor

2024-04-18T10:17:59.6166667+00:00

Hi @HR
Welcome to Microsoft Q&A platform and thanks for posting your question here.

when you scan a subset of views from 1 schema to a collection in Microsoft Purview, views that have been added to a different collection using the same schema are removed because the scan is configured to only scan the selected views for that specific collection. This means that if you have the same views in multiple collections, and you scan a subset of those views for one collection, the views that are not included in that subset will be removed from that collection.

To avoid this issue and ensure that the views are not removed from either collection, you can create separate scans for each collection and configure each scan to scan only the required views. By doing so, you can ensure that each scan is only scanning the views that are required for that specific collection, and the views that are not included in the scan will not be removed from that collection.

By creating separate scans for each collection, you can optimize the scan performance by scanning only the required views for each collection. This can help reduce the time and resources required for the scan.

Reference
https://learn.microsoft.com/en-us/purview/how-to-create-and-manage-collections

Hope this helps. Do let us know if you any further queries.

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.
Please sign in to rate this answer.
HR 0 Reputation points

2024-04-18T10:37:34.9033333+00:00

Thank you Harishga for your comment.

Just to confirm:

I have registered the Azure Synapse Analytics source only once, to a parent collection e.g. parentcollection1

I have 2 collections e.g. collection1 and collection2

I have created a scan rule set for each collection e.g.

scanruleset1 for collection1

scanruleset2 for collection2

I have created 2 scans e.g.

scan1 using scanruleset1 to scan in assets (views) to collection1

scan2 using scanruleset2 to scan in assets (views) to collection2

The views I've specified in scanruleset1 are not specified in scanruleset2.

The views come from the same schema e.g. schema_views1

When I scan in the views to collection1 and collection2 there is no overlap, views that are added to collection1 are not in collection2 and vice versa.

During overnight scans for scan1 and scan2, I can see in the run history this error message if this helps 'A single datasource could only be registered once, please check if some of the datasources within the expansion scan has already been covered by other expansion scans/ registeed datasources'

HR 0 Reputation points

2024-04-18T11:35:03.8566667+00:00

Also, just to confirm the scan rule sets are custom scan rule sets with ignore patterns for each specifying the views to bring in.

scanruleset1 ignore patterns example:

^mssql://server-production-synapse01-ondemand.sql.azuresynapse.net/datalake/(?!.(?:schema_views1)).$ - This specifies to bring in views only within the schema 'schema_views1' If I don't include this then I ingest all the views from other schemas e.g. schema_views2, schema_views3

^mssql://server-production-synapse01-ondemand.sql.azuresynapse.net/datalake/schema_views1/(?!.(?:view1)\b).$ - This specifies to only bring in view1 from 'schema_views1'

scanruleset2 ignore patterns example:

^mssql://server-production-synapse01-ondemand.sql.azuresynapse.net/datalake/(?!.(?:schema_views1)).$ - This specifies to bring in views only within the schema 'schema_views1'

^mssql://server-production-synapse01-ondemand.sql.azuresynapse.net/datalake/schema_views1/(?!.(?:view2)\b).$ - This specifies to only bring in view2 from 'schema_views1'

The Azure Synapse Workspace 'server-production-synapse01.azuresynapse.net:443', Azure Synapse Serverless SQL Schema 'schema_views1' and Azure Synapse Serverless SQL Database 'datalake' are removed from collection1 and scanned into collection2.

Harishga 3,425 Reputation points Microsoft Vendor

2024-04-19T04:54:58.7566667+00:00

HR

The error message you are seeing during overnight scans for scan1 and scan2, 'A single datasource could only be registered once, please check if some of the datasources within the expansion scan has already been covered by other expansion scans/ registered datasources,' indicates that the same data source has been registered multiple times, which is not allowed. This could be because the same data source is being scanned by both scan1 and scan2, or because the data source is being scanned by another expansion scan.

Regarding your question about scanning views from the same Azure Synapse Analytics schema to multiple collections, it seems like you are using ignore patterns in your scan rule sets to specify which views to bring in. If you want to ensure that the views are not removed from either collection, you can include the views in both scan rule sets. Alternatively, you can create a single scan rule set that includes all the views you want to bring in and use it for both scans.

I hope this information helps you. Let me know if you have any further questions or concerns.

HR 0 Reputation points

2024-04-23T17:33:44.2466667+00:00

Hi Harishga,

I can see now that the error message related to scans that were done using old ignore patterns whereby I was ignoring datasets. When new views and schemas were added to Synapse these were ingested causing the error.

I attempted to mitigate against this by creating new ignore patterns that were negated essentially saying bring in x, y, z instead.

The issue of assets disappearing after scanning in different views to different collections from the same schema is still happening. The views I'm scanning in to each collection are different for each collection (but come from the same schema) and do not overlap. I just want to scan in different views to different collections without them being removed.

Thanks for your comments so far Harishga. :)

Harishga 3,425 Reputation points Microsoft Vendor

2024-04-24T07:04:20.33+00:00

HR
If the issue of assets disappearing after scanning in different views to different collections from the same schema is still happening, even after creating separate scans for each collection and specifying different views to scan in each scan, there could be other factors at play.

One possible reason for this could be that the scans are being run concurrently, and the assets are being removed by one scan before the other scan has a chance to scan them in. To avoid this issue, you can stagger the scans so that they are not running concurrently.

For example, you can schedule scan1 to run at a specific time, and then schedule scan2 to run after scan1 has completed. By doing so, you can ensure that each scan is running independently and not interfering with the other scan.

Another possible reason for this could be that the scans are not configured correctly. To ensure that the scans are configured correctly, you can review the scan rule sets and ignore patterns to ensure that they are correctly specifying the views to bring in. You can also review the scan logs to see if there are any errors or warnings that could be causing the issue.

I hope this information helps you. Let me know if you have any further questions or concerns.

HR 0 Reputation points

2024-04-24T12:49:56.63+00:00

Hi Harishga,

Thank you. The scans were running concurrently. I've staggered the times and will check tomorrow to see if this fixes the issue. Thank you very much for your help with this. :)

HR 0 Reputation points

2024-04-25T10:34:39.1933333+00:00

Hi Harishga,

Unfortunately the assets are still being removed. I've staggered the scans by 30 minutes (each takes up to 4 minutes to complete).

I've double checked that the assets scanned in to both collections from the same schema are different, there is no overlap.

I'd also like to note that when I'm scanning in assets from different schemas to different collections, they are working fine. It is when I'm scanning in assets from the same schema to different collections, the issue arises. Is there a bug with Synapse Serverless schemas?

Thank you.
Sign in to comment