ConnectException org.fluentd.logger.sender.RawSocketSender with Azure Monitor Integration

Question

When we enabled Azure Monitor on our HDInsight cluster (https://learn.microsoft.com/en-us/azure/hdinsight/log-analytics-migration) we got the following exception when running Spark jobs:

22/04/06 12:15:29 ERROR RawSocketSender [MdsLoggerSenderThread]: org.fluentd.logger.sender.RawSocketSender
java.net.ConnectException: Connection refused (Connection refused)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:607)
at org.fluentd.logger.sender.RawSocketSender.connect(RawSocketSender.java:85)
at org.fluentd.logger.sender.RawSocketSender.reconnect(RawSocketSender.java:94)
at org.fluentd.logger.sender.RawSocketSender.flush(RawSocketSender.java:193)
at org.fluentd.logger.sender.RawSocketSender.send(RawSocketSender.java:184)
at org.fluentd.logger.sender.RawSocketSender.emit(RawSocketSender.java:149)
at org.fluentd.logger.sender.RawSocketSender.emit(RawSocketSender.java:131)
at org.fluentd.logger.sender.RawSocketSender.emit(RawSocketSender.java:126)
at org.fluentd.logger.FluentLogger.log(FluentLogger.java:101)
at org.fluentd.logger.FluentLogger.log(FluentLogger.java:86)
at com.microsoft.mdsdclient.MessageSendingRunnable$1.call(Unknown Source)
at com.microsoft.mdsdclient.MessageSendingRunnable$1.call(Unknown Source)
at com.microsoft.mdsdclient.RetryUtil.retry(Unknown Source)
at com.microsoft.mdsdclient.RetryUtil.retry(Unknown Source)
at com.microsoft.mdsdclient.MessageSendingRunnable.sendMessage(Unknown Source)
at com.microsoft.mdsdclient.MessageSendingRunnable.run(Unknown Source)
at java.lang.Thread.run(Thread.java:748)

Our cluster is inside a virtual network, but there is nothing blocking access to the log analytics workspace.

Note this error looks similar: https://learn.microsoft.com/en-us/answers/questions/766880/can-not-import-packages-i-azure-synapse.html

Answer 1

Hello anonymous user,

As this confirmed as the product-bug and the resolution shared on the support ticket.

This issue is due to an uninitialized variable bug in azure-mdsd 1.16.0-build.master.343.

Run the below command on all the nodes to fix the socket issue (First test it on preprod cluster and then try it on prod cluster) and let us know the status.

sudo su  
wget https://packages.microsoft.com/repos/azurecore/pool/main/a/azure-mdsd/azure-mdsd_1.18.0-build.master.385_x86_64.deb && dpkg -i azure-mdsd_1.18.0-build.master.385_x86_64.deb  

Hope this will help. Please let us know if any further queries.

------------------------------

• Please don't forget to click on or upvote button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
• Want a reminder to come back and check responses? Here is how to subscribe to a notification
• If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators