Metadata driven Etl data lineage on purview

Question

Hi Guys,

Our Azure Data Factory uses meta data drive etl frame work something similar to below

https://learn.microsoft.com/en-us/azure/data-factory/copy-data-tool-metadata-driven

But more customized than above,

When we used purview to data catalog/ data lineage it failed to recognize same goes with other data Governance tools, any better options available?

Regards,
Navin

Answer 1

Hello @Anonymous ,

Thanks for the question and using MS Q&A platform.

The problem with metadata driven ETLs (and our ADF based interpretation is no different) is that the “content” of the pipeline is determined at runtime. By extension, the same goes for the lineage. In other words what the pipeline does is only known when the pipeline runs.

As per the conversation with internal team it may be possible only through the use of the purview APIs. So, inside the metadata driven pipeline, you may have to add extra activities that would call the APIs and dynamically which would create the lineage at runtime.

Below are few related helpful articles published by Microsoft professionals which might give you some hints to try out alternates for creating custom lineage for meta data drive ETLs:

• Azure Purview REST API, Part 3: Custom Lineage
• Use Azure Purview’s REST APIs for creating custom lineage
• Create custom lineage in Azure Purview
• A Python SDK for Azure Purview and Apache Atlas

But if you have any specific feedback to improve the functionality or a feature request for Azure Purview product, I would recommend you to please log it in IDEAS forum and do share the link once it is posted so that we can take it forward to relevant product owners. Here is the IDEAS forum: https://feedback.azure.com/d365community/forum/82d7bddb-fb24-ec11-b6e6-000d3a4f07b8

Hope this will help. Please let us know if any further queries.

------------------------------

• Please don't forget to click on or upvote button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
• Want a reminder to come back and check responses? Here is how to subscribe to a notification
• If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators