Cluster configuration error

Shambhu Rai 1,406

Hi Expert,

I am getting error in cluster configuration in databricks cluster creation

java.lang.NoClassDefFoundError: org/apache/hadoop/hive/metastore/api/NoSuchObjectException

spark.hadoop.javax.jdo.option.ConnectionDriverName com.microsoft.sqlserver.jdbc.SQLServerDriver spark.hadoop.javax.jdo.option.ConnectionURL z1.db.windows.net,1433;Initial Catalog=Hivemetastore;TrustServerCertificate=False;Connection Timeout=30;Authentication="Active Directory Default";

Skip this one if <hive-version> is 0.13.x. spark.conf.set("spark.databricks.delta.schema.autoMerge.enabled ","true") spark.databricks.delta.preview.enabled true spark.hadoop.javax.jdo.option.ConnectionUserName test spark.conf.set("spark.databricks.parquet.schema.autoMerge.enabled ","true") datanucleus.fixedDatastore false spark.hadoop.javax.jdo.option.ConnectionPassword "dd"

spark.sql("SET spark.databricks.delta.schema.autoMerge.enabled = true") hive.metastore.schema.verification.record.version to enable datanucleus.autoCreateSchema true

#spark.sql.hive.metastore.jars to builtin spark.sql.hive.metastore.jars set to maven hive.metastore.schema.verification false spark.sql.hive.metastore.version =1.2.0 #spark.sql.hive.metastore.version 2.3.0

is there anything i am missing in configuration

Vahid Ghafarpour 20,480 Reputation points

2023-07-21T18:32:56.9+00:00

Please check the compatibility of the Hive Metastore version with your Databricks cluster. Ensure that your version is supported and compatible with the Databricks environment. The versions of the Hive Metastore and Databricks components need to be aligned to prevent compatibility issues.
Vahid Ghafarpour 20,480 Reputation points

2023-07-21T18:33:22.1533333+00:00

The error might occur if some necessary JAR files are missing or incorrectly specified. Check if the required JARs are available and correctly set in the cluster configuration.
Shambhu Rai 1,406 Reputation points

2023-07-22T04:29:57.13+00:00

My current available version is 3.0.1 which is not compatible what to do which jar files required
PRADEEPCHEEKATLA-MSFT 84,856 Reputation points Microsoft Employee

2023-07-24T07:08:39.9966667+00:00
@Shambhu Rai - Thanks for the question and using MS Q&A platform.

It seems like you are facing an error while configuring the Databricks cluster. The error message indicates that the class org/apache/hadoop/hive/metastore/api/NoSuchObjectException is not found. This error usually occurs when the Hive metastore is not configured correctly.

To resolve this issue, you can try the following steps:

To resolve this issue, you can try the following steps:

telnet <metastore-host> <metastore-port>

Replace <metastore-host> and <metastore-port> with the actual values.

Make sure that the Hive metastore jars are included in the classpath of the Databricks cluster. You can add the following configuration to the cluster:

spark.sql.hive.metastore.jars=<path-to-hive-jars>

Replace <path-to-hive-jars> with the actual path to the Hive metastore jars.

Check if the hive.metastore.schema.verification property is set to false. You can add the following configuration to the cluster:

hive.metastore.schema.verification=false

Make sure that the JDBC driver for the metastore database is included in the classpath of the Databricks cluster. You can add the following configuration to the cluster:

spark.driver.extraClassPath=<path-to-jdbc-driver> spark.executor.extraClassPath=<path-to-jdbc-driver>

Replace <path-to-jdbc-driver> with the actual path to the JDBC driver.

Once you have made these changes, try creating the Databricks cluster again and see if the issue is resolved. If you still face the issue, please provide more details about the error message and the steps you have taken so far.

Hope this helps. Do let us know if you any further queries.

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.
Shambhu Rai 1,406 Reputation points

2023-07-24T09:28:14.52+00:00
added now getting new error while creating table

here is sql query

%sql CREATE TABLE test ( EMPCODE DECIMAL(10,0) ) LOCATION '/user/hive/warehouse/new1.db';

Error in SQL statement: AnalysisException: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to fetch table student_external2. Exception thrown when executing query : SELECT DISTINCT 'org.apache.hadoop.hive.metastore.model.MTable' AS NUCLEUS_TYPE,A0.CREATE_TIME,A0.LAST_ACCESS_TIME,A0.OWNER,A0.RETENTION,A0.IS_REWRITE_ENABLED,A0.TBL_NAME,A0.TBL_TYPE,A0.TBL_ID FROM TBLS A0 LEFT OUTER JOIN DBS B0 ON A0.DB_ID = B0.DB_ID WHERE A0.TBL_NAME = ? AND B0."NAME" = ?
PRADEEPCHEEKATLA-MSFT 84,856 Reputation points Microsoft Employee

2023-07-25T05:07:21.7866667+00:00

@Shambhu Rai - It seems like you are facing an error while creating a table in Hive. The error message indicates that there is an issue with the table student_external2. However, I don't see this table being used in the SQL query that you have provided.

To help you better, could you please provide more information about the error? Specifically, could you please provide the complete error message along with the stack trace? This will help me understand the issue better and provide you with an appropriate solution.
PRADEEPCHEEKATLA-MSFT 84,856 Reputation points Microsoft Employee

2023-07-27T08:23:53.42+00:00

@Shambhu Rai - Just checking in to see if the below answer helped. If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.
Shambhu Rai 1,406 Reputation points

2023-07-27T08:49:29.53+00:00

sir

when we create sql or any create statment it should store in db but it is not happening
PRADEEPCHEEKATLA-MSFT 84,856 Reputation points Microsoft Employee

2023-07-27T09:19:01.4+00:00

@Shambhu Rai - I agree that this issue looks strange and I wasn't able to reproduce this issue. If you have a support plan could you please file a support ticket for deeper investigation and do share the SR# with us?
Shambhu Rai 1,406 Reputation points

2023-07-27T09:20:55.0333333+00:00

sir, please add this code in compute cluster .. it is pointing to due to default hive metastore and not storing any data to azure sql server
PRADEEPCHEEKATLA-MSFT 84,856 Reputation points Microsoft Employee

2023-07-31T08:06:24.1433333+00:00

@Shambhu Rai - Please share the code snippet which you are trying alond with screenshot?
Shambhu Rai 1,406 Reputation points

2023-07-31T08:36:00.8266667+00:00

Hive metastore is already created by somone in azure on top of that when i am pointing to azure sql server it goes to default path in data option where as it is not storing any data to azure sql server database
PRADEEPCHEEKATLA-MSFT 84,856 Reputation points Microsoft Employee

2023-07-31T09:43:43.78+00:00

@Shambhu Rai - Please share the above requested details to investigate further. OR If you have a support plan could you please file a support ticket for deeper investigation and do share the SR# with us?

Share via

Cluster configuration error