Spark Job Failure

Varun Singh 1 Reputation point
2022-11-14T23:39:48.35+00:00

Hi

I am currently running a spark job which has suddenly started failing to give the below error

260228-image.png

We are using the configured Job clusters on Azure Data factory and hence the Job is triggered from ADF. This is the job Cluster Configuration we are using
260283-image.png

Not sure why this error is being thrown. Can you please help with this?

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
1,943 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,627 questions
0 comments

1 answer

Sort by: Most helpful
Most helpful Newest Oldest
  1. BhargavaGunnam-MSFT 26,331 Reputation points Microsoft Employee
    2022-11-15T20:38:04.393+00:00

    Hello @Varun Singh ,

    Welcome to the MS Q&A platform.

    As per this blog, this error occurs when there is no initialized Metastore database for the Hive.

    Do you have any external Metastore setup? if yes, could you please provide more details on it?

    Is this intermittent or happening every time?

    Can you please try restarting the cluster and see if it fixes the issue?