Databricks and Collibra

Question

All,

Has anyone tried to connect Collibra to read the meta-data from storage and lineage from the notebooks? The link seems to be indicating for meta-data only.

Thanks,
grajee

https://productresources.collibra.com/docs/collibra/latest/Content/Catalog/RegisterDataSource/JDBC/ref_db-connection-details-collibra-provided-driver.htm?catalog-connector-details=databricks

Answer 1

Hello @Gopinath Rajee ,

Thanks for the question and using MS Q&A platform.

We haven't tried to connect Collibra to read the meta-data from storage and lineage from the notebooks. I had reached out to the internal team to get more information on your ask.

As per my experience - you can use Microsoft Purview to extract the lineage from databricks.

Microsoft Purview team released an open source solution accelerator to extract lineage from Databricks and ingest it into Microsoft Purview: microsoft/Purview-ADB-Lineage-Solution-Accelerator: A connector to ingest Azure Databricks lineage into Microsoft Purview (github.com)

This solution accelerator, together with the OpenLineage project, provides a connector that will transfer lineage metadata from Spark operations in Azure Databricks to Microsoft Purview, allowing you to see a table-level lineage graph. It supports Delta, Azure SQL, Data Lake Gen 2, and more.

Hope this will help. Please let us know if any further queries.

------------------------------

• Please don't forget to click on or upvote button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is howd
• Want a reminder to come back and check responses? Here is how to subscribe to a notification
• If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators