Azure IoT Hub Python device constantly crashing

Question

Azure IoT Hub Python device constantly crashing

CogniTech 20

Hi all. We have an existing program that uses Intel Neural Compute stick to classify the stream from a USB camera and send messages to a broker. Previously we used our own MQTT broker and this device would run for months with no issues. We have swapped out the MQTT broker for IoT hub and it constantly crashes rendering the device unresponsive and the only solution is to restart the device, run the program again and it all repeats. This is quite concerning considering we have previously run this exact same software prior to the addition of Azure with absolutely no issues. The main program will check the provisioning state and then connect to the IoT hub, two threads are created, one for the reading of the camera stream and one for the streaming of the frames to a local web server where the stream can be monitored.

When it doesn't crash the speed of the stream is massively reduced, where as with our own MQTT broker the feed was near real-time, now we have replaced that with IoT Hub the stream is about 20 seconds behind, very jittery, and constantly freezes. We only send a message every 2 minutes so only introducing Azure is causing this. Errors like this constantly happen: {'_id': '27af74bb-5d6b-4933-b67a-0e42be9afc70', 'Data': [{'Sensor': 'Camera', 'Data': {'Dected': '0'}}], 'State': {'CPU': '24.1', 'Memory': '15.3', 'Diskspace': '41.9', 'CPUTemp': '59.887', 'IP': '192.168.1.97', 'MAC': 'dc:a6:32:75:89:a1'}, 'Location': {'Longitude': '-3.0833', 'Latitude': '53.1667'}} WARNING:azure.iot.device.common.pipeline.pipeline_stages_base:ConnectionStateStage: DisconnectEvent received while in unexpected state - ConnectionState.DISCONNECTED WARNING:azure.iot.device.common.handle_exceptions:Exception caught in background thread. Unable to handle. WARNING:azure.iot.device.common.handle_exceptions:['azure.iot.device.common.transport_exceptions.ConnectionDroppedError: Unexpected disconnection\n'] WARNING:azure.iot.device.common.handle_exceptions:Exception caught in background thread. Unable to handle. WARNING:azure.iot.device.common.handle_exceptions:['azure.iot.device.common.transport_exceptions.ConnectionDroppedError: Unexpected disconnection\n'] WARNING:azure.iot.device.common.handle_exceptions:Exception caught in background thread. Unable to handle. WARNING:azure.iot.device.common.handle_exceptions:['azure.iot.device.common.transport_exceptions.ConnectionFailedError\n'] WARNING:azure.iot.device.common.handle_exceptions:Exception caught in background thread. Unable to handle. WARNING:azure.iot.device.common.handle_exceptions:['azure.iot.device.common.transport_exceptions.ConnectionFailedError\n'] WARNING:azure.iot.device.common.handle_exceptions:Exception caught in background thread. Unable to handle. WARNING:azure.iot.device.common.handle_exceptions:['azure.iot.device.common.transport_exceptions.ConnectionFailedError\n'] WARNING:azure.iot.device.common.handle_exceptions:Exception caught in background thread. Unable to handle. WARNING:azure.iot.device.common.handle_exceptions:['azure.iot.device.common.transport_exceptions.ConnectionFailedError\n'] Message sent to IntelliConnect IoT We would like to find a solution for this issue if possible.

CogniTech 20

Without Azure none of this happens which is bizarre:

Exception in thread Thread-9:
Traceback (most recent call last):
  File "/home/pi/.local/lib/python3.7/site-packages/azure/iot/device/common/mqtt_transport.py", line 397, in connect
    host=self._hostname, port=8883, keepalive=self._keep_alive
  File "/home/pi/.local/lib/python3.7/site-packages/paho/mqtt/client.py", line 914, in connect
    return self.reconnect()
  File "/home/pi/.local/lib/python3.7/site-packages/paho/mqtt/client.py", line 1044, in reconnect
    sock = self._create_socket_connection()
  File "/home/pi/.local/lib/python3.7/site-packages/paho/mqtt/client.py", line 3685, in _create_socket_connection
    return socket.create_connection(addr, timeout=self._connect_timeout, source_address=source)
  File "/usr/lib/python3.7/socket.py", line 707, in create_connection
    for res in getaddrinfo(host, port, 0, SOCK_STREAM):
  File "/usr/lib/python3.7/socket.py", line 748, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -3] Temporary failure in name resolution

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/pi/.local/lib/python3.7/site-packages/azure/iot/device/iothub/sync_clients.py", line 33, in handle_result
    return callback.wait_for_completion()
  File "/home/pi/.local/lib/python3.7/site-packages/azure/iot/device/common/evented_callback.py", line 69, in wait_for_completion
    raise self.exception
  File "/home/pi/.local/lib/python3.7/site-packages/azure/iot/device/common/pipeline/pipeline_stages_mqtt.py", line 206, in _run_op
    self.transport.connect(password=password)
  File "/home/pi/.local/lib/python3.7/site-packages/azure/iot/device/common/mqtt_transport.py", line 419, in connect
    raise exceptions.ConnectionFailedError() from e
azure.iot.device.common.transport_exceptions.ConnectionFailedError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/lib/python3.7/threading.py", line 917, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.7/threading.py", line 865, in run
    self._target(*self._args, **self._kwargs)
  File "/home/pi/IntelliConnectExamples/RPI/4/facial-recognition/imports/read.py", line 50, in run
    self.intelliconn.send_message(label)
  File "/home/pi/IntelliConnectExamples/RPI/4/facial-recognition/imports/intelliconnect.py", line 76, in send_message
    self.client.send_message(msg)
  File "/home/pi/.local/lib/python3.7/site-packages/azure/iot/device/patch_documentation.py", line 60, in send_message
    return super(IoTHubDeviceClient, self).send_message(message)
  File "/home/pi/.local/lib/python3.7/site-packages/azure/iot/device/iothub/sync_clients.py", line 346, in send_message
    handle_result(callback)
  File "/home/pi/.local/lib/python3.7/site-packages/azure/iot/device/iothub/sync_clients.py", line 37, in handle_result
    raise exceptions.ConnectionFailedError("Could not connect to IoTHub") from e
azure.iot.device.exceptions.ConnectionFailedError: Could not connect to IoTHub

WARNING:azure.iot.device.common.handle_exceptions:Exception caught in background thread.  Unable to handle.
WARNING:azure.iot.device.common.handle_exceptions:['azure.iot.device.common.transport_exceptions.ConnectionFailedError\n']
WARNING:azure.iot.device.common.handle_exceptions:Exception caught in background thread.  Unable to handle.
WARNING:azure.iot.device.common.handle_exceptions:['azure.iot.device.common.transport_exceptions.ConnectionFailedError\n']
WARNING:azure.iot.device.common.handle_exceptions:Exception caught in background thread.  Unable to handle.
WARNING:azure.iot.device.common.handle_exceptions:['azure.iot.device.common.transport_exceptions.ConnectionFailedError\n']
WARNING:azure.iot.device.common.handle_exceptions:Exception caught in background thread.  Unable to handle.
WARNING:azure.iot.device.common.handle_exceptions:['azure.iot.device.common.transport_exceptions.ConnectionFailedError\n']
WARNING:azure.iot.device.common.handle_exceptions:Exception caught in background thread.  Unable to handle.
WARNING:azure.iot.device.common.handle_exceptions:['azure.iot.device.common.transport_exceptions.ConnectionFailedError\n']
WARNING:azure.iot.device.common.handle_exceptions:Exception caught in background thread.  Unable to handle.
WARNING:azure.iot.device.common.handle_exceptions:['azure.iot.device.common.transport_exceptions.ConnectionFailedError\n']

CogniTech 20 Reputation points

2024-02-06T16:52:02.25+00:00

Crashes seem to have gotten better for some reason, no code changes. Frames are still 20 -30 seconds behind in comparison to near real-time, would like to address this issue. Will report back if crashing starts again., but any explanation why the introduction of Azure has created a 30 second delay?

1 answer

Your answer

CogniTech 20 Reputation points

2024-02-06T16:52:02.25+00:00

Crashes seem to have gotten better for some reason, no code changes. Frames are still 20 -30 seconds behind in comparison to near real-time, would like to address this issue. Will report back if crashing starts again., but any explanation why the introduction of Azure has created a 30 second delay?

Answer 1

Sander van de Velde | MVP 36,761 MVP Volunteer Moderator

Hello @CogniTech ,

welcome to this moderated Azure community forum.

From the messages you share, many errors are related to connecting and disconnecting.

Azure IoT Hub communicates over MQTT with a limited set of predefined topics due the tight relationship with the DeviceTwin.

I recommend testing the connection first without the camera part to be sure the connection itself works as expected. If needed, this blog post explains how to connect with python using the SDK.

Are you sure the DeviceClient tries to keep the connection open or is it connecting and disconnecting for each message? Have you tried MQTT over websockets too (less depending on MQTT related firewall rules)?

As an alternative, you could try the new Azure MQTT broker based on Azure EventGrid. A video about how to use it can be found here.

If the response helped, do "Accept Answer". If it doesn't work, please let us know the progress. All community members with similar issues will benefit by doing so. Your contribution is highly appreciated.

CogniTech 20 Reputation points

2024-02-07T01:08:31.77+00:00

Not sure if it was a bad day on the server but this project is complete and has been running all day with no issues.
CogniTech 20 Reputation points

2024-02-07T01:09:22.6033333+00:00

Sorry just seen your message. Thanks for getting back. The errors seemed to just stop and we have been running the device most of the day wth no issues now.
Sander van de Velde | MVP 36,761 Reputation points MVP Volunteer Moderator

2024-02-07T08:30:27.6333333+00:00

I understand this is not an ideal situation but good to hear it works again.

Perhaps the suggestions and links above help you to get a better understanding of the stability.

If the response helped, do "Accept Answer". If it doesn't work, please let us know the progress. All community members with similar issues will benefit by doing so. Your contribution is highly appreciated.

Share via

Azure IoT Hub Python device constantly crashing

1 answer

Your answer