Execute Azure Databricks SQL script in Azure data factory

Jatinder Luthra 130 Reputation points
2023-11-27T21:57:58.5+00:00

Hello,

I am trying to create databricks dataset. When database.schema name is provided, it displays list of tables to choose from.
image

When click on Preview data, it throws error as below:
User's image

Any idea about this behavior?

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,330 questions
0 comments No comments
{count} votes

Accepted answer
  1. Bjarne Meyn 80 Reputation points
    2024-01-26T09:54:09.9+00:00

    Hi @Jatinder Luthra Hey, we ran into the same problem! We were trying to read from a specific database (catalogue) in Databricks, but kept getting this annoying 'Table not found' error. Turns out, the Azure Databricks Delta Lake Dataset was trying to pull the table from the hive_metastore, instead of the database (catalogue) we actually defined. I found a little workaround. If you leave the database and table properties in the Azure Databricks Delta Lake Dataset blank, and then use the linked service in a copy activity to define the query, you can pull the data from a different catalogue than the hive_metastore. User's image

    User's image

    User's image Hope this gets your problem sorted!

    1 person found this answer helpful.
    0 comments No comments

2 additional answers

Sort by: Most helpful
  1. PRADEEPCHEEKATLA 90,541 Reputation points
    2023-11-28T09:11:56.6833333+00:00

    @Jatinder Luthra - Thanks for the question and using MS Q&A platform.

    It seems like there is an issue with the table or view 'dcs.default'.'person' which cannot be found. The error message suggests that you should verify the spelling and correctness of the schema and catalog. If you did not qualify the name with a schema, verify the current_schema() output, or qualify the name with the correct schema and catalog.

    You can try the following steps to resolve the issue:

    1. Check if the table or view exists in the specified schema and catalog.
    2. Verify that the schema and catalog names are spelled correctly.
    3. Check if the user has the necessary permissions to access the table or view.
    4. Try qualifying the name with the correct schema and catalog.

    For more details, refer to Error handling in Azure Databricks and TABLE_OR_VIEW_NOT_FOUND error class.

    I hope this helps! Let me know if you have any further questions.


  2. Jatinder Luthra 130 Reputation points
    2024-01-26T14:35:27.6266667+00:00

    Thanks @Bjarne Meyn
    This is helpful. We also end up doing something similar.

    0 comments No comments

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.