@S, Santhosh M I apologize for the confusion earlier. You are correct that the Hive Metastore does not have the capability for lineage visualization.
Regarding your specific query about establishing lineage between two schemas in Azure Databricks Unity Catalog, it is possible that the lineage is not being captured because the data for tables within the second schema is loaded from the first schema using Scala code.
To capture lineage in this scenario, you can try the following steps:
- Make sure that the tables in both schemas are registered in the Azure Databricks Unity Catalog.
- Ensure that the Scala code used to load data from the first schema to the second schema is using Spark DataFrame APIs.
- Check if the Scala code is creating temporary views for the tables in both schemas. If not, you can create temporary views for the tables in both schemas using the
createOrReplaceTempView
method. - Once the temporary views are created, you can use the
spark.sql
method to execute SQL queries that join the tables in both schemas. This will ensure that the lineage is captured in the Azure Databricks Unity Catalog.
If you are still facing issues in capturing lineage between the two schemas, I would recommend you to check the logs and error messages to identify the root cause of the issue. You can also refer to the Azure Databricks documentation for more information on establishing lineage in Azure Databricks Unity Catalog.
I hope this helps! Let me know if you have any further questions.