question

VarunSharma-4299 avatar image
0 Votes"
VarunSharma-4299 asked PRADEEPCHEEKATLA-MSFT commented

Reverse lookup between two HDI cluster fails


Reverse lookup between two clusters fail, we have created script action to update the host files on our spark worker nodes. When scaling up the new worker nodes in spark have appropriate host entries to connect to hbase but when spark is scaled down there are some hdi scripts running system level which resets the host files.

azure-hdinsight
· 3
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hello @VarunSharma-4299,

Thanks for the question and using MS Q&A platform.

When you says "Reverse lookup between two HDI clusters fails" - are you experiencing any error message?
If yes, could you please share the complete error message or the screenshot?

0 Votes 0 ·

The documentation states

Spark cron job – checks if a Spark scaling occurred and if cluster is secure. If so, edit /etc/hosts to include HBase IP mapping stored locally

https://docs.microsoft.com/en-us/azure/hdinsight/hdinsight-using-spark-query-hbase

Hbase we get
Caused by: KrbException: Server not found in Kerberos database (7)
at sun.security.krb5.KrbTgsRep.<init>(KrbTgsRep.java:70)
at sun.security.krb5.KrbTgsReq.getReply(KrbTgsReq.java:226)
at sun.security.krb5.KrbTgsReq.sendAndGetCreds(KrbTgsReq.java:237)
at sun.security.krb5.internal.CredentialsUtil.serviceCredsSingle(CredentialsUtil.java:400)
at sun.security.krb5.internal.CredentialsUtil.serviceCreds(CredentialsUtil.java:287)
at sun.security.krb5.internal.CredentialsUtil.serviceCreds(CredentialsUtil.java:263)
at sun.security.krb5.internal.CredentialsUtil.acquireServiceCreds(CredentialsUtil.java:118)
at sun.security.krb5.Credentials.acquireServiceCreds(Credentials.java:490)
at sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:695)
... 29 more

Spark
WARN Client: Exception encountered while connecting to the server : javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]

0 Votes 0 ·

Hello @VarunSharma-4299,

I see the warning message and not the error message, could you please share the complete stacktrace of the error message?

0 Votes 0 ·

0 Answers