Azure SQL Database
An Azure relational database service.
5,567 questions
Cluster configuration error
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(140.8, 75%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESR%3C/text%3E%3C/svg%3E)
Shambhu Rai
1,411
Reputation points
Hi Expert,
I am getting error in cluster configuration in databricks cluster creation
java.lang.NoClassDefFoundError: org/apache/hadoop/hive/metastore/api/NoSuchObjectException
spark.hadoop.javax.jdo.option.ConnectionDriverName com.microsoft.sqlserver.jdbc.SQLServerDriver spark.hadoop.javax.jdo.option.ConnectionURL z1.db.windows.net,1433;Initial Catalog=Hivemetastore;TrustServerCertificate=False;Connection Timeout=30;Authentication="Active Directory Default";
Skip this one if <hive-version> is 0.13.x. spark.conf.set("spark.databricks.delta.schema.autoMerge.enabled ","true") spark.databricks.delta.preview.enabled true spark.hadoop.javax.jdo.option.ConnectionUserName test spark.conf.set("spark.databricks.parquet.schema.autoMerge.enabled ","true") datanucleus.fixedDatastore false spark.hadoop.javax.jdo.option.ConnectionPassword "dd"
spark.sql("SET spark.databricks.delta.schema.autoMerge.enabled = true") hive.metastore.schema.verification.record.version to enable datanucleus.autoCreateSchema true
#spark.sql.hive.metastore.jars to builtin spark.sql.hive.metastore.jars set to maven hive.metastore.schema.verification false spark.sql.hive.metastore.version =1.2.0 #spark.sql.hive.metastore.version 2.3.0
is there anything i am missing in configuration
{count} votes
-
Vahid Ghafarpour 21,075 Reputation points2023-07-21T18:32:56.9+00:00 Please check the compatibility of the Hive Metastore version with your Databricks cluster. Ensure that your version is supported and compatible with the Databricks environment. The versions of the Hive Metastore and Databricks components need to be aligned to prevent compatibility issues.
-
Vahid Ghafarpour 21,075 Reputation points
2023-07-21T18:33:22.1533333+00:00 The error might occur if some necessary JAR files are missing or incorrectly specified. Check if the required JARs are available and correctly set in the cluster configuration.
-
Shambhu Rai 1,411 Reputation points
2023-07-22T04:29:57.13+00:00 My current available version is 3.0.1 which is not compatible what to do which jar files required
-
PRADEEPCHEEKATLA-MSFT 88,301 Reputation points Microsoft Employee2023-07-24T07:08:39.9966667+00:00 @Shambhu Rai - Thanks for the question and using MS Q&A platform.
It seems like you are facing an error while configuring the Databricks cluster. The error message indicates that the class
org/apache/hadoop/hive/metastore/api/NoSuchObjectExceptionis not found. This error usually occurs when the Hive metastore is not configured correctly.To resolve this issue, you can try the following steps:
To resolve this issue, you can try the following steps:
telnet <metastore-host> <metastore-port>Replace
<metastore-host>and<metastore-port>with the actual values.Make sure that the Hive metastore jars are included in the classpath of the Databricks cluster. You can add the following configuration to the cluster:
spark.sql.hive.metastore.jars=<path-to-hive-jars>Replace
<path-to-hive-jars>with the actual path to the Hive metastore jars.Check if the
hive.metastore.schema.verificationproperty is set tofalse. You can add the following configuration to the cluster:hive.metastore.schema.verification=falseMake sure that the JDBC driver for the metastore database is included in the classpath of the Databricks cluster. You can add the following configuration to the cluster:
spark.driver.extraClassPath=<path-to-jdbc-driver> spark.executor.extraClassPath=<path-to-jdbc-driver>Replace
<path-to-jdbc-driver>with the actual path to the JDBC driver.Once you have made these changes, try creating the Databricks cluster again and see if the issue is resolved. If you still face the issue, please provide more details about the error message and the steps you have taken so far.
Hope this helps. Do let us know if you any further queries.
If this answers your query, do click
Accept AnswerandYesfor was this answer helpful. And, if you have any further query do let us know. -
Shambhu Rai 1,411 Reputation points
2023-07-24T09:28:14.52+00:00 added now getting new error while creating table
here is sql query
%sql CREATE TABLE test ( EMPCODE DECIMAL(10,0) ) LOCATION '/user/hive/warehouse/new1.db';Error in SQL statement: AnalysisException: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to fetch table student_external2. Exception thrown when executing query : SELECT DISTINCT 'org.apache.hadoop.hive.metastore.model.MTable' AS NUCLEUS_TYPE,A0.CREATE_TIME,A0.LAST_ACCESS_TIME,A0.OWNER,A0.RETENTION,A0.IS_REWRITE_ENABLED,A0.TBL_NAME,A0.TBL_TYPE,A0.TBL_ID FROM TBLS A0 LEFT OUTER JOIN DBS B0 ON A0.DB_ID = B0.DB_ID WHERE A0.TBL_NAME = ? AND B0."NAME" = ?
-
PRADEEPCHEEKATLA-MSFT 88,301 Reputation points Microsoft Employee
2023-07-25T05:07:21.7866667+00:00 @Shambhu Rai - It seems like you are facing an error while creating a table in Hive. The error message indicates that there is an issue with the table
student_external2. However, I don't see this table being used in the SQL query that you have provided.To help you better, could you please provide more information about the error? Specifically, could you please provide the complete error message along with the stack trace? This will help me understand the issue better and provide you with an appropriate solution.
-
PRADEEPCHEEKATLA-MSFT 88,301 Reputation points Microsoft Employee
2023-07-27T08:23:53.42+00:00 @Shambhu Rai - Just checking in to see if the below answer helped. If this answers your query, do click
Accept AnswerandYesfor was this answer helpful. And, if you have any further query do let us know. -
Shambhu Rai 1,411 Reputation points
2023-07-27T08:49:29.53+00:00 sir
when we create sql or any create statment it should store in db but it is not happening
-
PRADEEPCHEEKATLA-MSFT 88,301 Reputation points Microsoft Employee
2023-07-27T09:19:01.4+00:00 @Shambhu Rai - I agree that this issue looks strange and I wasn't able to reproduce this issue. If you have a support plan could you please file a support ticket for deeper investigation and do share the SR# with us?
-
Shambhu Rai 1,411 Reputation points
2023-07-27T09:20:55.0333333+00:00 sir, please add this code in compute cluster .. it is pointing to due to default hive metastore and not storing any data to azure sql server
-
PRADEEPCHEEKATLA-MSFT 88,301 Reputation points Microsoft Employee
2023-07-31T08:06:24.1433333+00:00 @Shambhu Rai - Please share the code snippet which you are trying alond with screenshot?
-
Shambhu Rai 1,411 Reputation points
2023-07-31T08:36:00.8266667+00:00 Hive metastore is already created by somone in azure on top of that when i am pointing to azure sql server it goes to default path in data option where as it is not storing any data to azure sql server database
-
PRADEEPCHEEKATLA-MSFT 88,301 Reputation points Microsoft Employee
2023-07-31T09:43:43.78+00:00 @Shambhu Rai - Please share the above requested details to investigate further. OR If you have a support plan could you please file a support ticket for deeper investigation and do share the SR# with us?
Sign in to comment
Sign in to answer