I am using Azure ML for real-time machine learning. I have installed the Kafka server, but I am having a connection issue when trying to create a topic using the below line of code. I received the following warning: WARN [AdminClient clientId=adminclient-1] Connection to node -1 (localhost/127.0.0.1:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient). I appreciate your help. !./kafka_2.13-3.3.2/bin/kafka-topics.sh --create --topic amids-train --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1

Hi Ghada, The warning message you received suggests that the Kafka broker may not be available or is not running on the specified address and port. Here are some steps you can follow to troubleshoot the issue: Verify that the Kafka broker is running: You can check if the Kafka broker is running by using the following command in a new terminal window: ./kafka_2.13-3.3.2/bin/kafka-server-start.sh ./kafka_2.13-3.3.2/config/server.properties Verify that the address and port are correct: Make sure that the address and port specified in the bootstrap-server parameter are correct and that there are no firewall or network configuration issues preventing you from connecting to the broker. Check the Kafka logs for errors: Check the Kafka logs to see if there are any error messages that could help identify the issue. You can find the Kafka logs in the logs directory of your Kafka installation. Try using a different topic name: It's possible that the topic name you're using is already in use or is invalid. Try using a different topic name to see if that resolves the issue. I hope these steps help you resolve the issue. Let me know if you have any further questions!

Azure ML and Kafka Server Connection Issue

Accepted answer

Zeeshan Nasir Bajwa 646 Reputation points Student Ambassador

2023-04-07T05:07:41.48+00:00
Hi Ghada, The warning message you received suggests that the Kafka broker may not be available or is not running on the specified address and port. Here are some steps you can follow to troubleshoot the issue:

Verify that the Kafka broker is running: You can check if the Kafka broker is running by using the following command in a new terminal window: ./kafka_2.13-3.3.2/bin/kafka-server-start.sh ./kafka_2.13-3.3.2/config/server.properties

Verify that the address and port are correct: Make sure that the address and port specified in the bootstrap-server parameter are correct and that there are no firewall or network configuration issues preventing you from connecting to the broker.

Check the Kafka logs for errors: Check the Kafka logs to see if there are any error messages that could help identify the issue. You can find the Kafka logs in the logs directory of your Kafka installation.

Try using a different topic name: It's possible that the topic name you're using is already in use or is invalid. Try using a different topic name to see if that resolves the issue.

I hope these steps help you resolve the issue. Let me know if you have any further questions!
Please sign in to rate this answer.

1 person found this answer helpful.
Ghada 31 Reputation points

2023-04-07T05:43:38.5366667+00:00

Thank you very much for your comment. I verified that the server is running on a different port than the one I was using (9092). I changed the port to the 2181, but it still not working. I changed the topic but noticed that the connection was refused. I am using the Azure ML workspace to run the notebook code.

WARN Session 0x0 for sever localhost/127.0.0.1:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException. (org.apache.zookeeper.ClientCnxn) java.net.ConnectException: Connection refused.

Zeeshan Nasir Bajwa 646 Reputation points Student Ambassador

2023-04-07T06:09:17.8466667+00:00

Based on the error message you provided, it seems like the connection to the ZooKeeper server is being refused. ZooKeeper is a distributed coordination service used by Kafka to manage its cluster. When you start a Kafka broker, it connects to the ZooKeeper ensemble to register itself and participate in the cluster. Here are some steps you can follow to troubleshoot the issue: null

bashCopy code ./kafka_2.13-3.3.2/bin/zookeeper-server-start.sh ./kafka_2.13-3.3.2/config/zookeeper.properties

nullnullnullnull

I hope these steps help you resolve the issue. Let me know if you have any further questions!

Zeeshan Nasir Bajwa 646 Reputation points Student Ambassador

2023-04-07T06:09:36.87+00:00

Based on the error message you provided, it seems like the connection to the ZooKeeper server is being refused. ZooKeeper is a distributed coordination service used by Kafka to manage its cluster. When you start a Kafka broker, it connects to the ZooKeeper ensemble to register itself and participate in the cluster. Here are some steps you can follow to troubleshoot the issue: Verify that the ZooKeeper server is running: Make sure that the ZooKeeper server is running and accessible from the machine where you're running the Kafka command. You can check if the ZooKeeper server is running by using the following command: bash Copy code ./kafka_2.13-3.3.2/bin/zookeeper-server-start.sh ./kafka_2.13-3.3.2/config/zookeeper.properties Verify that the ZooKeeper server is running on the correct port: By default, ZooKeeper listens on port 2181. Make sure that the port number you're using is correct. Check the ZooKeeper logs for errors: Check the ZooKeeper logs to see if there are any error messages that could help identify the issue. You can find the ZooKeeper logs in the logs directory of your Kafka installation. Verify that the ZooKeeper ensemble is configured correctly: If you're running a Kafka cluster with multiple brokers, make sure that the ZooKeeper ensemble is configured correctly in your server.properties file. Check for firewall or network configuration issues: Make sure that there are no firewall or network configuration issues preventing you from connecting to the ZooKeeper server. I hope these steps help you resolve the issue. Let me know if you have any further questions!

Ghada 31 Reputation points

2023-04-07T07:55:34.8233333+00:00

Thank you both for the great comments. I was able to make some progress except for the timed out error while executing topic command. The Timed out waiting for a node assignment.

Ghada 31 Reputation points

2023-04-07T11:07:49.9633333+00:00

I was able to create the topics successfully. I invoked the terminal to start the zookeeper and bootstrap server using the same commands except that I added the option -daemon: ./kafka_2.13-3.3.2/bin/zookeeper-server-start.sh -daemon ./kafka_2.13-3.3.2/config/zookeeper.properties. In addition, I used the default port for the bootstrap server 9092.
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

Share via

Azure ML and Kafka Server Connection Issue

0 additional answers

Your answer