If "Metadata discovery" has status as "completed with execption" will it affect classification results? please explain

Question

If "Metadata discovery" has status as "completed with execption" will it affect classification results? please explain

Dhanoordaran V 150

Question 1:

In "Run detail" under scan status, if "Metadata discovery" has status as "completed with exception" will it affect classification results? please explain the logic behind it

Question 2:

When examining the scan error log, it is mostly because of two errors :"parquet java invocation error" and "not a valid parquet file", how to address that.

Kindly help to address them.

Thanks

Accepted answer

0 additional answers

Your answer

Answer 1

J N S S Kasyap 1,625 Microsoft External Staff

Hi @Dhanoordaran V

In "Run detail" under scan status, if "Metadata discovery" has status as "completed with exception" will it affect classification results?

It could affect classification results, depending on the nature of the exception and how critical the metadata is to the classification process.
The "completed with exception" status indicates that the metadata discovery process finished but encountered issues (exceptions) that prevented it from fully or correctly completing its task. For example, some files might not have been processed, or certain metadata might be missing or corrupted.

Exceptions during metadata discovery suggest that the system couldn’t fully prepare the data for classification. For instance:

A missing schema might lead to incorrect feature mapping.
Unprocessed files might reduce the dataset size, skewing results.
Partial metadata might confuse the classifier about data context.

The severity depends on the amount of data affected; if only a small subset of files triggered exceptions, the impact might be negligible.

When examining the scan error log, it is mostly because of two errors :"parquet java invocation error" and "not a valid parquet file", how to address that.

The below Microsoft Documentation will help for the above question

https://learn.microsoft.com/en-us/azure/data-factory/connector-troubleshoot-parquet

I hope this information helps. Please do let us know if you have any further queries.

Kindly consider upvoting the comment if the information provided is helpful. This can assist other community members in resolving similar issues.

Thank you.

Dhanoordaran V 150 Reputation points

2025-04-04T09:39:19.1533333+00:00

@J N S S Kasyap , Thanks a lot for the input
Dhanoordaran V 150 Reputation points

2025-04-07T11:28:53.88+00:00
Hi @J N S S Kasyap , I have a follow up query in "scan run details".

Is the "assets classified" in the run detail will gets added up on "total assets classified" in classification insights dashboard.? please explain

How we do relate classification summary on scan vs on the classification report
J N S S Kasyap 1,625 Reputation points Microsoft External Staff

2025-04-07T21:43:15.19+00:00

@Dhanoordaran V

Is the "assets classified" in the run detail will gets added up on "total assets classified" in classification insights dashboard.?

Yes, the “Assets classified” shown in the scan run detail can contribute to the “Total assets classified” in the classification insights dashboard, but only under specific conditions. The scan run detail reflects the number of assets classified during that particular scan, while the dashboard aggregates results across all scans.

If the assets were newly classified during the scan and had not been previously classified, they will typically be included in the dashboard's total. However, if those assets were already classified in a prior scan or if classification failed for some assets due to scan issues, they may not increase the dashboard’s count.
The classification insights dashboard shows deduplicated, cumulative totals across the data estate. It may also be filtered by asset type, time range, or classification rule. If the same asset is scanned multiple times, it will not be double-counted unless a new classification is applied. Therefore, not all classified assets from a scan will necessarily reflect as a direct increase in the classification insights unless they meet these criteria.

Please refer the below Microsoft Document for more details
https://learn.microsoft.com/en-us/purview/data-map-classification

How we do relate classification summary on scan vs on the classification report

The classification summary in a scan shows results for that specific scan run only including how many assets were discovered and classified during that run. It gives a point-in-time view and includes details such as scan status, exceptions (if any), and how many assets were successfully classified in that execution.
The classification report in the insights dashboard, on the other hand, provides a cumulative and aggregated view across all scans over time. It includes all successfully classified assets in the catalog and supports filtering by asset type, classification rule, scan source, and date range. While the scan summary shows immediate results from one scan, the insights dashboard consolidates and visualizes the overall classification posture across your environment. If a scan identifies new classified assets, those will be reflected in the dashboard, provided they were not already classified earlier.

I hope this information helps. Please do let us know if you have any further queries.
Dhanoordaran V 150 Reputation points

2025-04-08T04:03:04.95+00:00

Thanks a lot for the reply.

Share via

If "Metadata discovery" has status as "completed with execption" will it affect classification results? please explain

0 additional answers

Your answer