ConnectException org.fluentd.logger.sender.RawSocketSender with Azure Monitor Integration

Thomas Schouten 61 Reputation points
2022-04-06T13:39:00.22+00:00

When we enabled Azure Monitor on our HDInsight cluster (https://learn.microsoft.com/en-us/azure/hdinsight/log-analytics-migration) we got the following exception when running Spark jobs:

22/04/06 12:15:29 ERROR RawSocketSender [MdsLoggerSenderThread]: org.fluentd.logger.sender.RawSocketSender
java.net.ConnectException: Connection refused (Connection refused)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:607)
at org.fluentd.logger.sender.RawSocketSender.connect(RawSocketSender.java:85)
at org.fluentd.logger.sender.RawSocketSender.reconnect(RawSocketSender.java:94)
at org.fluentd.logger.sender.RawSocketSender.flush(RawSocketSender.java:193)
at org.fluentd.logger.sender.RawSocketSender.send(RawSocketSender.java:184)
at org.fluentd.logger.sender.RawSocketSender.emit(RawSocketSender.java:149)
at org.fluentd.logger.sender.RawSocketSender.emit(RawSocketSender.java:131)
at org.fluentd.logger.sender.RawSocketSender.emit(RawSocketSender.java:126)
at org.fluentd.logger.FluentLogger.log(FluentLogger.java:101)
at org.fluentd.logger.FluentLogger.log(FluentLogger.java:86)
at com.microsoft.mdsdclient.MessageSendingRunnable$1.call(Unknown Source)
at com.microsoft.mdsdclient.MessageSendingRunnable$1.call(Unknown Source)
at com.microsoft.mdsdclient.RetryUtil.retry(Unknown Source)
at com.microsoft.mdsdclient.RetryUtil.retry(Unknown Source)
at com.microsoft.mdsdclient.MessageSendingRunnable.sendMessage(Unknown Source)
at com.microsoft.mdsdclient.MessageSendingRunnable.run(Unknown Source)
at java.lang.Thread.run(Thread.java:748)

Our cluster is inside a virtual network, but there is nothing blocking access to the log analytics workspace.

Note this error looks similar: https://learn.microsoft.com/en-us/answers/questions/766880/can-not-import-packages-i-azure-synapse.html

Azure HDInsight
Azure HDInsight
An Azure managed cluster service for open-source analytics.
201 questions
{count} votes

Accepted answer
  1. PRADEEPCHEEKATLA-MSFT 82,196 Reputation points Microsoft Employee
    2022-05-02T05:54:27.74+00:00

    Hello anonymous user,

    As this confirmed as the product-bug and the resolution shared on the support ticket.

    This issue is due to an uninitialized variable bug in azure-mdsd 1.16.0-build.master.343.

    Run the below command on all the nodes to fix the socket issue (First test it on preprod cluster and then try it on prod cluster) and let us know the status.

    sudo su  
    wget https://packages.microsoft.com/repos/azurecore/pool/main/a/azure-mdsd/azure-mdsd_1.18.0-build.master.385_x86_64.deb && dpkg -i azure-mdsd_1.18.0-build.master.385_x86_64.deb  
    

    Hope this will help. Please let us know if any further queries.

    ------------------------------

    • Please don't forget to click on 130616-image.png or upvote 130671-image.png button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
    • Want a reminder to come back and check responses? Here is how to subscribe to a notification
    • If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators
    1 person found this answer helpful.

0 additional answers

Sort by: Most helpful