Share via

How to restart the Kafka services on HDInsight Kafka Cluster

Kommana, Swathi 0 Reputation points
2024-06-18T09:55:39.4166667+00:00

Im trying to restart the kafka services which were configured on HDInsight kafka cluster as below. But they are getting timeout. Please advise how to proceed.

n0-kafkah:/usr/hdp/4.1.18.1/kafka/bin$ sudo ./kafka-server-start.sh /etc/kafka/conf/server.properties 
[2024-06-18 09:14:37,237] INFO Registered kafka:type=kafka.Log4jController MBean (kafka.utils.Log4jControllerRegistration$)
[2024-06-18 09:14:38,095] INFO starting (kafka.server.KafkaServer)
[2024-06-18 09:14:38,096] INFO Connecting to zookeeper on localhost:2181 (kafka.server.KafkaServer)
[2024-06-18 09:14:38,136] INFO [ZooKeeperClient] Initializing a new session to localhost:2181. (kafka.zookeeper.ZooKeeperClient)
[2024-06-18 09:14:38,154] INFO Client environment:zookeeper.version=3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf, built on 03/06/2019 16:18 GMT (org.apache.zookeeper.ZooKeeper)
[2024-06-18 09:14:38,154] INFO Client environment:host.name=hn0-kafkah.3gwtyg0clpyedcmxttgaz3o5sa.bx.internal.cloudapp.net (org.apache.zookeeper.ZooKeeper)
[2024-06-18 09:14:38,154] INFO Client environment:java.version=1.8.0_382 (org.apache.zookeeper.ZooKeeper)
[2024-06-18 09:14:38,154] INFO Client environment:java.vendor=Temurin (org.apache.zookeeper.ZooKeeper)
[2024-06-18 09:14:38,154] INFO Client environment:java.home=/usr/lib/jvm/temurin-8-jdk-amd64/jre (org.apache.zookeeper.ZooKeeper)
[2024-06-18 09:14:38,154] INFO Client environment:java.class.path=/usr/hdp/4.1.18.1/kafka/bin/../libs/activation-1.1.1.jar:/usr/hdp/4.1.18.1/kafka/bin/../libs/aopalliance-repackaged-2.5.0-b42.jar:/usr/hdp/4.1.18.1/kafka/bin/../libs/argparse4j-0.7.0.jar:/usr/hdp/4.1.18.1/kafka/bin/../libs/audience-annotations-0.5.0.jar:/usr/hdp/4.1.18.1/kafka/bin/../libs/caffeine-2.6.1.jar:/usr/hdp/4.1.18.1/kafka/bin/../libs/commons-lang3-3.8.1.jar:/usr/hdp/4.1.18.1/kafka/bin/../libs/connect-api-2.1.1.4.1.18.1.jar:/usr/hdp/4.1.18.1/kafka/bin/../libs/connect-basic-auth-extension-2.1.1.4.1.18.1.jar:/usr/hdp/4.1.18.1/kafka/bin/../libs/connect-file-2.1.1.4.1.18.1.jar:/usr/hdp/4.1.18.1/kafka/bin/../libs/connect-json-2.1.1.4.1.18.1.jar:/usr/hdp/4.1.18.1/kafka/bin/../libs/connect-runtime-2.1.1.4.1.18.1.jar:/usr/hdp/4.1.18.1/kafka/bin/../libs/connect-transforms-2.1.1.4.1.18.1.jar:/usr/hdp/4.1.18.1/kafka/bin/../libs/fluent-logger-0.3.4.jar:/usr/hdp/4.1.18.1/kafka/bin/../libs/guava-20.0.jar:/usr/hdp/4.1.18.1/kafka/bin/../libs/hdinsight-kafka3v0-1.1.jar:/usr/hdp/4.1.18.1/kafka/bin/../libs/hk2-api-2.5.0-b42.jar:/usr/hdp/4.1.18.1/kafka/bin/../libs/hk2-locator-2.5.0-b42.jar:/usr/hdp/4.1.18.1/kafka/bin/../libs/hk2-utils-2.5.0-b42.jar:/usr/hdp/4.1.18.1/kafka/bin/../libs/jackson-annotations-2.10.0.jar:/usr/hdp/4.1.18.1/kafka/bin/../libs/jackson-core-2.10.0.jar:/usr/hdp/4.1.18.1/kafka/bin/../libs/jackson-databind-2.10.0.jar:/usr/hdp/4.1.18.1/kafka/bin/../libs/jackson-jaxrs-base-2.10.0.jar:/usr/hdp/4.1.18.1/kafka/bin/../libs/jackson-jaxrs-json-provider-2.10.0.jar:/usr/hdp/4.1.18.1/kafka/bin/../libs/jackson-module-jaxb-annotations-2.10.0.jar:/usr/hdp/4.1.18.1/kafka/bin/../libs/jakarta.activation-api-1.2.1.jar:/usr/hdp/4.1.18.1/kafka/bin/../libs/jakarta.xml.bind-api-2.3.2.jar:/usr/hdp/4.1.18.1/kafka/bin/../libs/javassist-3.22.0-CR2.jar:/usr/hdp/4.1.18.1/kafka/bin/../libs/javax.annotation-api-1.2.jar:/usr/hdp/4.1.18.1/kafka/bin/../libs/javax.inject-1.jar:/usr/hdp/4.1.18.1/kafka/bin/../libs/javax.inject-2.5.0-b42.jar:/usr/hdp/4.1.18.1/kafka/bin/../libs/javax.servlet-api-3.1.0.jar:/usr/hdp/4.1.18.1/kafka/bin/../libs/javax.ws.rs-api-2.1.1.jar:/usr/hdp/4.1.18.1/kafka/bin/../libs/javax.ws.rs-api-2.1.jar:/usr/hdp/4.1.18.1/kafka/bin/../libs/jaxb-api-2.3.0.jar:/usr/hdp/4.1.18.1/kafka/bin/../libs/jersey-client-2.27.jar:/usr/hdp/4.1.18.1/kafka/bin/../libs/jersey-common-2.27.jar:/usr/hdp/4.1.18.1/kafka/bin/../libs/jersey-container-servlet-2.27.jar:/usr/hdp/4.1.18.1/kafka/bin/../libs/jersey-container-servlet-core-2.27.jar:/usr/hdp/4.1.18.1/kafka/bin/../libs/jersey-hk2-2.27.jar:/usr/hdp/4.1.18.1/kafka/bin/../libs/jersey-media-jaxb-2.27.jar:/usr/hdp/4.1.18.1/kafka/bin/../libs/jersey-server-2.27.jar:/usr/hdp/4.1.18.1/kafka/bin/../libs/jetty-client-9.4.20.v20190813.jar:/usr/hdp/4.1.18.1/kafka/bin/../libs/jetty-continuation-9.4.20.v20190813.jar:/usr/hdp/4.1.18.1/kafka/bin/../libs/jetty-http-9.4.20.v20190813.jar:/usr/hdp/4.1.18.1/kafka/bin/../libs/jetty-io-9.4.20.v20190813.jar:/usr/hdp/4.1.18.1/kafka/bin/../libs/jetty-security-9.4.20.v20190813.jar:/usr/hdp/4.1.18.1/kafka/bin/../libs/jetty-server-9.4.20.v20190813.jar:/usr/hdp/4.1.18.1/kafka/bin/../libs/jetty-servlet-9.4.20.v20190813.jar:/usr/hdp/4.1.18.1/kafka/bin/../libs/jetty-servlets-9.4.20.v20190813.jar:/usr/hdp/4.1.18.1/kafka/bin/../libs/jetty-util-9.4.20.v20190813.jar:/usr/hdp/4.1.18.1/kafka/bin/../libs/jetty-util-ajax-9.4.20.v20190813.jar:/usr/hdp/4.1.18.1/kafka/bin/../libs/jopt-simple-5.0.4.jar:/usr/hdp/4.1.18.1/kafka/bin/../libs/json-20090211.jar:/usr/hdp/4.1.18.1/kafka/bin/../libs/json-simple-1.1.jar:/usr/hdp/4.1.18.1/kafka/bin/../libs/kafka-clients-2.1.1.4.1.18.1.jar:/usr/hdp/4.1.18.1/kafka/bin/../libs/kafka-ganglia-2.1.1.4.1.18.1-sources.jar:/usr/hdp/4.1.18.1/kafka/bin/../libs/kafka-ganglia-2.1.1.4.1.18.1.jar:/usr/hdp/4.1.18.1/kafka/bin/../libs/kafka-log4j-appender-2.1.1.4.1.18.1.jar:/usr/hdp/4.1.18.1/kafka/bin/../libs/kafka-streams-2.1.1.4.1.18.1.jar:/usr/hdp/4.1.18.1/kafka/bin/../libs/kafka-streams-examples-2.1.1.4.1.18.1.jar:/usr/hdp/4.1.18.1/kafka/bin/../libs/kafka-streams-scala_2.11-2.1.1.4.1.18.1.jar:/usr/hdp/4.1.18.1/kafka/bin/../libs/kafka-streams-test-utils-2.1.1.4.1.18.1.jar:/usr/hdp/4.1.18.1/kafka/bin/../libs/kafka-tools-2.1.1.4.1.18.1.jar:/usr/hdp/4.1.18.1/kafka/bin/../libs/kafka_2.11-2.1.1.4.1.18.1-sources.jar:/usr/hdp/4.1.18.1/kafka/bin/../libs/kafka_2.11-2.1.1.4.1.18.1.jar:/usr/hdp/4.1.18.1/kafka/bin/../libs/lz4-java-1.5.0.jar:/usr/hdp/4.1.18.1/kafka/bin/../libs/maven-artifact-3.6.0.jar:/usr/hdp/4.1.18.1/kafka/bin/../libs/mdsdclient-1.0.jar:/usr/hdp/4.1.18.1/kafka/bin/../libs/metrics-core-2.2.0.jar:/usr/hdp/4.1.18.1/kafka/bin/../libs/microsoft-log4j-etwappender-1.0.jar:/usr/hdp/4.1.18.1/kafka/bin/../libs/msgpack-0.6.8.jar:/usr/hdp/4.1.18.1/kafka/bin/../libs/osgi-resource-locator-1.0.1.jar:/usr/hdp/4.1.18.1/kafka/bin/../libs/plexus-utils-3.1.0.jar:/usr/hdp/4.1.18.1/kafka/bin/../libs/ranger-kafka-plugin-impl:/usr/hdp/4.1.18.1/kafka/bin/../libs/ranger-kafka-plugin-shim-1.2.0.4.1.18.1.jar:/usr/hdp/4.1.18.1/kafka/bin/../libs/ranger-plugin-classloader-1.2.0.4.1.18.1.jar:/usr/hdp/4.1.18.1/kafka/bin/../libs/reflections-0.9.11.jar:/usr/hdp/4.1.18.1/kafka/bin/../libs/reload4j-1.2.19.jar:/usr/hdp/4.1.18.1/kafka/bin/../libs/rocksdbjni-5.14.2.jar:/usr/hdp/4.1.18.1/kafka/bin/../libs/rolling_upgrade.sh:/usr/hdp/4.1.18.1/kafka/bin/../libs/scala-library-2.11.12.jar:/usr/hdp/4.1.18.1/kafka/bin/../libs/scala-logging_2.11-3.9.0.jar:/usr/hdp/4.1.18.1/kafka/bin/../libs/scala-reflect-2.11.12.jar:/usr/hdp/4.1.18.1/kafka/bin/../libs/slf4j-api-1.7.36.jar:/usr/hdp/4.1.18.1/kafka/bin/../libs/slf4j-reload4j-1.7.36.jar:/usr/hdp/4.1.18.1/kafka/bin/../libs/snappy-java-1.1.7.2.jar:/usr/hdp/4.1.18.1/kafka/bin/../libs/storm:/usr/hdp/4.1.18.1/kafka/bin/../libs/validation-api-1.1.0.Final.jar:/usr/hdp/4.1.18.1/kafka/bin/../libs/zkclient-0.11.jar:/usr/hdp/4.1.18.1/kafka/bin/../libs/zookeeper-3.4.14.jar:/usr/hdp/4.1.18.1/kafka/bin/../libs/zookeeper.jar:/usr/hdp/4.1.18.1/kafka/bin/../libs/zstd-jni-1.3.7-1.jar (org.apache.zookeeper.ZooKeeper)
[2024-06-18 09:14:38,155] INFO Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib (org.apache.zookeeper.ZooKeeper)
[2024-06-18 09:14:38,155] INFO Client environment:java.io.tmpdir=/tmp (org.apache.zookeeper.ZooKeeper)
[2024-06-18 09:14:38,155] INFO Client environment:java.compiler=<NA> (org.apache.zookeeper.ZooKeeper)
[2024-06-18 09:14:38,155] INFO Client environment:os.name=Linux (org.apache.zookeeper.ZooKeeper)
[2024-06-18 09:14:38,155] INFO Client environment:os.arch=amd64 (org.apache.zookeeper.ZooKeeper)
[2024-06-18 09:14:38,155] INFO Client environment:os.version=5.4.0-1129-azure (org.apache.zookeeper.ZooKeeper)
[2024-06-18 09:14:38,155] INFO Client environment:user.name=root (org.apache.zookeeper.ZooKeeper)
[2024-06-18 09:14:38,155] INFO Client environment:user.home=/root (org.apache.zookeeper.ZooKeeper)
[2024-06-18 09:14:38,155] INFO Client environment:user.dir=/usr/hdp/4.1.18.1/kafka/bin (org.apache.zookeeper.ZooKeeper)
[2024-06-18 09:14:38,162] INFO Initiating client connection, connectString=localhost:2181 sessionTimeout=6000 watcher=kafka.zookeeper.ZooKeeperClient$ZooKeeperClientWatcher$@4802796d (org.apache.zookeeper.ZooKeeper)
[2024-06-18 09:14:38,194] INFO [ZooKeeperClient] Waiting until connected. (kafka.zookeeper.ZooKeeperClient)
[2024-06-18 09:14:38,202] INFO Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
[2024-06-18 09:14:38,214] INFO Socket error occurred: localhost/127.0.0.1:2181: Connection refused (org.apache.zookeeper.ClientCnxn)
[2024-06-18 09:14:39,321] INFO Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
[2024-06-18 09:14:39,322] INFO Socket error occurred: localhost/127.0.0.1:2181: Connection refused (org.apache.zookeeper.ClientCnxn)
[2024-06-18 09:14:40,426] INFO Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
[2024-06-18 09:14:40,427] INFO Socket error occurred: localhost/127.0.0.1:2181: Connection refused (org.apache.zookeeper.ClientCnxn)
[2024-06-18 09:14:41,527] INFO Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
[2024-06-18 09:14:41,528] INFO Socket error occurred: localhost/127.0.0.1:2181: Connection refused (org.apache.zookeeper.ClientCnxn)
[2024-06-18 09:14:42,629] INFO Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
[2024-06-18 09:14:42,630] INFO Socket error occurred: localhost/127.0.0.1:2181: Connection refused (org.apache.zookeeper.ClientCnxn)
[2024-06-18 09:14:43,730] INFO Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
[2024-06-18 09:14:43,731] INFO Socket error occurred: localhost/127.0.0.1:2181: Connection refused (org.apache.zookeeper.ClientCnxn)
[2024-06-18 09:14:44,207] INFO [ZooKeeperClient] Closing. (kafka.zookeeper.ZooKeeperClient)
[2024-06-18 09:14:44,832] INFO Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
[2024-06-18 09:14:44,935] INFO Session: 0x0 closed (org.apache.zookeeper.ZooKeeper)
[2024-06-18 09:14:44,935] INFO EventThread shut down for session: 0x0 (org.apache.zookeeper.ClientCnxn)
[2024-06-18 09:14:44,971] INFO [ZooKeeperClient] Closed. (kafka.zookeeper.ZooKeeperClient)
[2024-06-18 09:14:44,975] ERROR Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
kafka.zookeeper.ZooKeeperClientTimeoutException: Timed out waiting for connection while in state: CONNECTING
        at kafka.zookeeper.ZooKeeperClient$$anonfun$kafka$zookeeper$ZooKeeperClient$$waitUntilConnected$1.apply$mcV$sp(ZooKeeperClient.scala:268)
        at kafka.zookeeper.ZooKeeperClient$$anonfun$kafka$zookeeper$ZooKeeperClient$$waitUntilConnected$1.apply(ZooKeeperClient.scala:264)
        at kafka.zookeeper.ZooKeeperClient$$anonfun$kafka$zookeeper$ZooKeeperClient$$waitUntilConnected$1.apply(ZooKeeperClient.scala:264)
        at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:251)
        at kafka.zookeeper.ZooKeeperClient.kafka$zookeeper$ZooKeeperClient$$waitUntilConnected(ZooKeeper
Azure HDInsight
Azure HDInsight
An Azure managed cluster service for open-source analytics.
212 questions

1 answer

Sort by: Most helpful
Most helpful Newest Oldest
  1. PRADEEPCHEEKATLA-MSFT 90,221 Reputation points Microsoft Employee
    2024-06-19T07:09:50.6333333+00:00

    @Kommana, Swathi - Thanks for the question and using MS Q&A platform.

    Have you tried restarting all option which in available in the Ambari portal?

    To restart Kafka, use the Restart button and select Restart All Affected. Confirm the restart, and then use the OK button after the operation has completed.

    Hope this helps. Do let us know if you any further queries.

    If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.

    0 comments

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.