Purview scan Synapse only shows 2 of 3 pipelines and several files in blob storage are not found

Question

Hi, I'm preparing a Purview demo for a customer on my free azure account.

I have a few issues with purview assets at the moment. Perhaps they are caused by my inexperience with the product.

1. One pipeline uses a copy data activity to read a csv file from a website and fill a table in a dedicated sql pool. The only asset discovered by purview is the table itself. The copy activity and pipeline are missing.
2. I have another pipeline which moves data from a csv file to a json file using a dataflow. When looking at the lineage of the dataflow itself it shows the input csv and output json file in the storage as an asset. It also shows in the "Recently accessed" list. However, the asset is not visible as a type "File" when scrolling the assets in the collection, only as a "Azure Data Lake Storage Gen2 Resource Set" . I do not understand why. Other files in the same folder are visible as File-assets
3. I renamed a copy activity in synapse studio and did a full scan, but the assets name remains the same ( other changes were processed into purview collection correctly ).

Scanning storage and synapse pools does not help. I tried full and incremental scans.

Any guidance would be greatly appreciated.

Regards Roel

Answer 1

Hello @Roel Evers

Thanks for sharing the Scenario on this forum!

Regarding your first issue, it's possible that the copy activity and pipeline are not being discovered by Purview because they are not considered assets in the same way that tables are. Create resource set pattern rules

Regarding your second issue, it's possible that the file is not being recognized as a "File" asset because it is part of a resource set.

Customizing resource set grouping using pattern rules

Regarding your third issue, it's possible that the asset name is being cached by Purview and needs to be refreshed. You can try clearing the cache.