Cannot Create or List any spark Database

Question

Cannot Create or List any spark Database

Alexander B. Hurtado 6

Im trying to create a spark database either using spark.sql("CREATE DATABASE dbname") or df.write.mode("overwrite").saveAsTable(dbname.tablename) and is giving me this error:

AnalysisException: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient;
Traceback (most recent call last):
File “/opt/spark/python/lib/pyspark.zip/pyspark/sql/session.py”, line 767, in sql
return DataFrame(self._jsparkSession.sql(sqlQuery), self._wrapped)
File “/opt/spark/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py”, line 1257, in call
answer, self.gateway_client, self.target_id, self.name)
File “/opt/spark/python/lib/pyspark.zip/pyspark/sql/utils.py”, line 75, in deco
raise AnalysisException(s.split(‘: ’, 1)[1], stackTrace)
pyspark.sql.utils.AnalysisException: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient;

It algo gives me the same error when I try to list the tables using: print(spark.catalog.listTables())

PRADEEPCHEEKATLA 91,866 Reputation points

2021-05-27T05:44:36.31+00:00
Hello @Alexander B. Hurtado ,

Welcome to the Microsoft Q&A platform.

In order to investigate further, could you please share the below details:

This could be a transient issue. Could you please retry and do let us know.

Are you facing this issue only specific to single Spark pool?

Could you please try to create a new spark pool and check if the problem exists?
Alexander B. Hurtado 6 Reputation points

2021-05-27T15:26:02.91+00:00

Hi, I have been trying for the last 3 days, same result.

I have tested with 3 different Spark Pools with different specs, all of them give me the same result.
PRADEEPCHEEKATLA 91,866 Reputation points

2021-05-28T05:47:49.66+00:00

Hello @Alexander B. Hurtado ,

Thanks for the confirmation.

We are reaching out to product team to get help on this issue and will update you as soon as we have a response.

Thank you for your patience.
PRADEEPCHEEKATLA 91,866 Reputation points

2021-05-31T12:16:50.133+00:00

Hello @Alexander B. Hurtado ,

Could you please share workspace name and region of the Azure Synapse Analytics?
Alexander B. Hurtado 6 Reputation points

2021-06-01T15:34:49.23+00:00

Hi @PRADEEPCHEEKATLA ,

Workspacename: occu-synapse-test

Region: US West
Alexander B. Hurtado 6 Reputation points

2021-06-01T15:36:18.943+00:00

Hi,

Workspace: occu-synapse-test
Region: US West

Thank you
PRADEEPCHEEKATLA 91,866 Reputation points

2021-06-02T03:32:38.153+00:00

Hello @Alexander B. Hurtado ,

Thanks for sharing the details.

I had shared these details with the product team to get help on this issue and will update you as soon as we have a response.

Thank you for your patience.

1 answer

Your answer

PRADEEPCHEEKATLA 91,866 Reputation points

2021-05-27T05:44:36.31+00:00

Hello @Alexander B. Hurtado ,

Welcome to the Microsoft Q&A platform.

In order to investigate further, could you please share the below details:

This could be a transient issue. Could you please retry and do let us know.

Are you facing this issue only specific to single Spark pool?

Could you please try to create a new spark pool and check if the problem exists?
Alexander B. Hurtado 6 Reputation points

2021-05-27T15:26:02.91+00:00

Hi, I have been trying for the last 3 days, same result.

I have tested with 3 different Spark Pools with different specs, all of them give me the same result.
PRADEEPCHEEKATLA 91,866 Reputation points

2021-05-28T05:47:49.66+00:00

Hello @Alexander B. Hurtado ,

Thanks for the confirmation.

We are reaching out to product team to get help on this issue and will update you as soon as we have a response.

Thank you for your patience.
PRADEEPCHEEKATLA 91,866 Reputation points

2021-05-31T12:16:50.133+00:00

Hello @Alexander B. Hurtado ,

Could you please share workspace name and region of the Azure Synapse Analytics?
Alexander B. Hurtado 6 Reputation points

2021-06-01T15:34:49.23+00:00

Hi @PRADEEPCHEEKATLA ,

Workspacename: occu-synapse-test

Region: US West
Alexander B. Hurtado 6 Reputation points

2021-06-01T15:36:18.943+00:00

Hi,

Workspace: occu-synapse-test
Region: US West

Thank you
PRADEEPCHEEKATLA 91,866 Reputation points

2021-06-02T03:32:38.153+00:00

Hello @Alexander B. Hurtado ,

Thanks for sharing the details.

I had shared these details with the product team to get help on this issue and will update you as soon as we have a response.

Thank you for your patience.

Answer 1

PRADEEPCHEEKATLA 91,866

Hello @Alexander B. Hurtado ,

We had fixed the issue for your workspace. Could you please check and confirm on the same?

This was one of the small set workspaces where we failed to enable SyMS. We created a hotfix from backend to resolve the issue.

Hope this helps. Do let us know if you any further queries.

---------------------------------------------------------------------------

Please "Accept the answer" if the information helped you. This will help us and others in the community as well.

Alexander B. Hurtado 6 Reputation points

2021-06-04T13:35:23.833+00:00

Hi @PRADEEPCHEEKATLA

Thanks for the help. Can you please explained a little bit about what happened, so we don't make the same mistake in the future?
PRADEEPCHEEKATLA 91,866 Reputation points

2021-06-07T01:56:23.627+00:00

Hello @Alexander B. Hurtado ,

It's was an issue from the backend, which we had fixed to resolve the issue.
PRADEEPCHEEKATLA 91,866 Reputation points

2021-06-08T03:29:51.097+00:00

Hello @Alexander B. Hurtado ,

Just checking in to see if the above answer helped. If this answers your query, do click Accept Answer and Up-Vote for the same. And, if you have any further query do let us know.

Share via

Cannot Create or List any spark Database

1 answer

Your answer