Error while trying to connect to on Prem impala from databricks to read a table using pyspark

Question

Error while trying to connect to on Prem impala from databricks to read a table using pyspark

29577539 0

I am using 11.3 lts runtime version of azure databricks and impala cluster version is 3.4.0-Snapshot.With this i have installed the impala jdbc version :2.6.4.1005 in the cluster and the code I am using as below:df=spark.read.format("jdbc").option("url","jdbc:impala://hostname:portname/database).option("dbtable","tblnm").option("user",usname).option("password",PWD).option('driver','com.cloudera.impala.jdbc41.Driver').load() after execution it's throwing error as Simaba Impalajdbcdriver 700110 Unexpected session error:Java.lang.NoclassDefFoundError and if we install some other version of impala and in option we pass option('driver','com.cloudera.impala.jdbc.Driver').load() it gives error as Communication link failure.Could you please suggest what might be causing the issue as the port is already opened and we are able to connect to impala through pyodbc.

PRADEEPCHEEKATLA 90,651 Reputation points Moderator

2023-06-22T06:18:39.1766667+00:00

@29577539 - Just checking in to see if the below answer helped. If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.
PRADEEPCHEEKATLA 90,651 Reputation points Moderator

2023-07-04T05:00:08.2233333+00:00

@29577539 - We received your feedback that the answer provided on the thread was not helpful.

Kindly let us know what we could have done better to improve the answer and make your engagement experience good. We are here to help you and strive to make your experience better and greatly value your feedback.

I have provided a detailed answer which has clear steps for which you are looking for. If you wish, you may re-surveying/rating for the engagement you received on the thread. Your feedback is very important to us.

Looking forward to you reply. Much appreciate your feedback!

Regards,

PRADEEPCHEEKATLA-MSFT

1 answer

Your answer

PRADEEPCHEEKATLA 90,651 Reputation points Moderator

2023-06-22T06:18:39.1766667+00:00

@29577539 - Just checking in to see if the below answer helped. If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.
PRADEEPCHEEKATLA 90,651 Reputation points Moderator

2023-07-04T05:00:08.2233333+00:00

@29577539 - We received your feedback that the answer provided on the thread was not helpful.

Kindly let us know what we could have done better to improve the answer and make your engagement experience good. We are here to help you and strive to make your experience better and greatly value your feedback.

I have provided a detailed answer which has clear steps for which you are looking for. If you wish, you may re-surveying/rating for the engagement you received on the thread. Your feedback is very important to us.

Looking forward to you reply. Much appreciate your feedback!

Regards,

PRADEEPCHEEKATLA-MSFT

Answer 1

PRADEEPCHEEKATLA 90,651 Moderator

@29577539 - Thanks for the question and using MS Q&A platform.

The error message you are seeing indicates that there is a communication issue between Databricks and your on-premises Impala instance. Here are some steps you can take to troubleshoot and resolve the issue:

Check network connectivity: Ensure that there is network connectivity between Databricks and your on-premises Impala instance. You can use tools like ping or telnet to test connectivity.
Check firewall settings: Ensure that the necessary ports are open in the firewall settings for your on-premises Impala instance. The default port for Impala is 21050, but this may vary depending on your configuration.
Check Impala configuration: Ensure that Impala is configured to allow connections from Databricks. You may need to add the IP address or hostname of the Databricks cluster to the Impala configuration.
Check JDBC driver version: Ensure that you are using the correct version of the JDBC driver for your Impala instance. You can download the latest version of the Cloudera Impala JDBC driver from the Cloudera website.
Check Databricks cluster configuration: Ensure that the Databricks cluster is configured to use the correct JDBC driver and connection settings for your Impala instance. You can configure these settings in the cluster configuration settings.
Check Impala logs: Check the Impala logs for any errors or warnings related to the connection issue. This may provide additional information on the cause of the issue.

By following these steps, you should be able to identify and resolve the communication issue between Databricks and your on-premises Impala instance.

And also, checkout the MS Q&A thread: How can i connect to my on premise Impala system from Azure databricks using python/pyspark code addressing similar issue.

Hope this helps. Do let us know if you any further queries.

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.

29577539 0 Reputation points

2023-06-22T06:29:00.5833333+00:00

Facing SSL issue even though the above steps were done
PRADEEPCHEEKATLA 90,651 Reputation points Moderator

2023-06-23T08:35:26.54+00:00

@29577539 - Could you please share the error message which you are facing?
29577539 0 Reputation points

2023-06-27T11:27:28.2933333+00:00

Hi Team,Still facing same issue no doubt the port for impala is already opened still getting same error unable to establish jdbc connection
29577539 0 Reputation points

2023-06-27T14:56:59.0333333+00:00

I am using 11.3 lts runtime version of azure databricks and impala cluster version is 3.4.0-Snapshot.With this i have installed the impala jdbc version :2.6.4.1005 in the cluster and the code I am using as below:df=spark.read.format("jdbc").option("url","jdbc:impala://hostname:portname/database).option("dbtable","tblnm").option("user",usname).option("password",PWD).option('driver','com.cloudera.impala.jdbc41.Driver').load() after execution it's throwing error as Simaba Impalajdbcdriver 700110 Unexpected session error:Java.lang.NoclassDefFoundError and if we install some other version of impala and in option we pass option('driver','com.cloudera.impala.jdbc.Driver').load() it gives error as Communication link failure.Could you please suggest what might be causing the issue as the port is already opened and we are able to connect to impala through pyodbc.
PRADEEPCHEEKATLA 90,651 Reputation points Moderator

2023-06-28T04:14:04.9633333+00:00

@29577539 - I agree that this issue looks strange and I wasn't able to reproduce this issue. If you have a support plan could you please file a support ticket for deeper investigation and do share the SR# with us? In case if you don't have a support plan please let us know here.
PRADEEPCHEEKATLA 90,651 Reputation points Moderator

2023-06-29T09:25:54.21+00:00

@29577539 - I agree that this issue looks strange and I wasn't able to reproduce this issue. If you have a support plan could you please file a support ticket for deeper investigation and do share the SR# with us?
29577539 0 Reputation points

2023-06-29T14:11:54.4+00:00

Hi the connection is getting established now but after reading the impala table the o/p is looking strange for example it is populating the column names in the rows rather than the row values and while trying to use display or show command it errors out with ClouderaJDBC 10140 ERROR CONVERTING VALUE TO INT .In short it's not able to read the data from impala using pyspark jdbc connection
PRADEEPCHEEKATLA 90,651 Reputation points Moderator

2023-07-04T04:58:59.85+00:00

@29577539 - Glad to know your issue has been resolved.

The scope of your question is different compared to the original scope of the question asked.

I would recommend creating a new thread on the same forum with as much details about your issue as possible. That would make sure that your issue has better visibility in the community.

Share via

Error while trying to connect to on Prem impala from databricks to read a table using pyspark

1 answer

Your answer