Azure IoT Edge
An Azure service that is used to deploy cloud workloads to run on internet of things (IoT) edge devices via standard containers.
534 questions
IoT Edge 1.2 connectivity check debugging
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(275.2, 65%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ER%3C/text%3E%3C/svg%3E)
randomrabbit
121
Reputation points
I've installed Edge Runtime 1.2 on Debian 11. I realize it is not a Tier 1 system but there are packages provided for it.
My Edge device successfully provisions to IoT Central (not Hub, but Central).
I'm not able to use this device as a gateway, I get an "Error in the IoTHub Client due to TLS exchanges" from my downstream device. It can successfully connect to a gateway running Edge Runtime 1.1 though, so I don't think there is any error on the downstream device. Here is the error:
INFO:azure.iot.device.common.mqtt_transport:Creating client for connecting using MQTT over TCP
INFO:azure.iot.device.iothub.aio.async_clients:Connecting to Hub...
INFO:azure.iot.device.common.mqtt_transport:Connect using port 8883 (TCP)
INFO:azure.iot.device.common.mqtt_transport:Forcing paho disconnect to prevent it from automatically reconnecting
INFO:azure.iot.device.common.pipeline.pipeline_stages_mqtt:transport.connect raised error
INFO:azure.iot.device.common.pipeline.pipeline_stages_mqtt:Traceback (most recent call last):
File "/Users/myuser/.venv/lib/python3.10/site-packages/azure/iot/device/common/mqtt_transport.py", line 390, i
n connect
rc = self._mqtt_client.connect(
File "/Users/myuser/.venv/lib/python3.10/site-packages/paho/mqtt/client.py", line 914, in connect
return self.reconnect()
File "/Users/myuser/.venv/lib/python3.10/site-packages/paho/mqtt/client.py", line 1073, in reconnect
sock.do_handshake()
File "/Users/myuser/.pyenv/versions/3.10.2/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/ssl.py", line 1341,
in do_handshake
self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: Hostname mismatch, certificate is not va
lid for 'my.valid.hostname'. (_ssl.c:997)
Here's my 'sudo iotedge check --verbose'
Configuration checks (aziot-identity-service)
---------------------------------------------
√ keyd configuration is well-formed - OK
√ certd configuration is well-formed - OK
√ tpmd configuration is well-formed - OK
√ identityd configuration is well-formed - OK
√ daemon configurations up-to-date with config.toml - OK
√ identityd config toml file specifies a valid hostname - OK
√ aziot-identity-service package is up-to-date - OK
√ host time is close to reference time - OK
√ preloaded certificates are valid - OK
√ keyd is running - OK
√ certd is running - OK
√ identityd is running - OK
√ read all preloaded certificates from the Certificates Service - OK
√ read all preloaded key pairs from the Keys Service - OK
√ ensure all preloaded certificates match preloaded private keys with the same ID - OK
Connectivity checks (aziot-identity-service)
--------------------------------------------
‼ host can connect to and perform TLS handshake with iothub AMQP port - Warning
Could not retrieve iothub_hostname from provisioning file.
Please specify the backing IoT Hub name using --iothub-hostname switch if you have that information.
Since no hostname is provided, all hub connectivity tests will be skipped.
‼ host can connect to and perform TLS handshake with iothub HTTPS / WebSockets port - Warning
Could not retrieve iothub_hostname from provisioning file.
Please specify the backing IoT Hub name using --iothub-hostname switch if you have that information.
Since no hostname is provided, all hub connectivity tests will be skipped.
‼ host can connect to and perform TLS handshake with iothub MQTT port - Warning
Could not retrieve iothub_hostname from provisioning file.
Please specify the backing IoT Hub name using --iothub-hostname switch if you have that information.
Since no hostname is provided, all hub connectivity tests will be skipped.
√ host can connect to and perform TLS handshake with DPS endpoint - OK
Configuration checks
--------------------
√ aziot-edged configuration is well-formed - OK
√ configuration up-to-date with config.toml - OK
√ container engine is installed and functional - OK
√ configuration has correct URIs for daemon mgmt endpoint - OK
√ aziot-edge package is up-to-date - OK
√ container time is close to host time - OK
‼ DNS server - Warning
Container engine is not configured with DNS server setting, which may impact connectivity to IoT Hub.
Please see https://aka.ms/iotedge-prod-checklist-dns for best practices.
You can ignore this warning if you are setting DNS server per module in the Edge deployment.
caused by: Could not open container engine config file /etc/docker/daemon.json
caused by: No such file or directory (os error 2)
‼ production readiness: logs policy - Warning
Container engine is not configured to rotate module logs which may cause it run out of disk space.
Please see https://aka.ms/iotedge-prod-checklist-logs for best practices.
You can ignore this warning if you are setting log policy per module in the Edge deployment.
caused by: Could not open container engine config file /etc/docker/daemon.json
caused by: No such file or directory (os error 2)
‼ production readiness: Edge Agent's storage directory is persisted on the host filesystem - Warning
The edgeAgent module is not configured to persist its /tmp/edgeAgent directory on the host filesystem.
Data might be lost if the module is deleted or updated.
Please see https://aka.ms/iotedge-storage-host for best practices.
‼ production readiness: Edge Hub's storage directory is persisted on the host filesystem - Warning
The edgeHub module is not configured to persist its /tmp/edgeHub directory on the host filesystem.
Data might be lost if the module is deleted or updated.
Please see https://aka.ms/iotedge-storage-host for best practices.
‼ Agent image is valid and can be pulled from upstream - Warning
skipping because of previous failures
√ proxy settings are consistent in aziot-edged, aziot-identityd, moby daemon and config.toml - OK
Connectivity checks
-------------------
‼ container on the default network can connect to upstream AMQP port - Warning
skipping because of previous failures
‼ container on the default network can connect to upstream HTTPS / WebSockets port - Warning
skipping because of previous failures
‼ container on the default network can connect to upstream MQTT port - Warning
skipping because of previous failures
‼ container on the IoT Edge module network can connect to upstream AMQP port - Warning
skipping because of previous failures
‼ container on the IoT Edge module network can connect to upstream HTTPS / WebSockets port - Warning
skipping because of previous failures
‼ container on the IoT Edge module network can connect to upstream MQTT port - Warning
skipping because of previous failures
How could I debug the "host can connect to and perform TLS handshake with iothub MQTT port" and "upstream MQTT port" issue, which I think is the culprint? Is the "Could not retrieve iothub_hostname from provisioning file." fine, since my gateway connects to IoT Central using symmetric key authentication?
Edit: The command
openssl s_client -connect mygateway.contoso.com:8883 -CAfile <CERTDIR>/certs/azure-iot-test-only.root.ca.cert.pem -showcerts
from https://learn.microsoft.com/en-us/azure/iot-edge/how-to-connect-downstream-device?view=iotedge-2020-11 seems to run fine from my downstream machine
{count} votes
-
António Sérgio Azevedo 7,661 Reputation points Microsoft Employee2022-04-28T12:03:52.603+00:00 @randomrabbit to be sure you are using the latest versions of the IoT Edge agent and IoT Edge hub images can you add in your manifest the three values of the version number to explicitly set the image version? See the releases here. The latest version today is 1.2.9:
"image": "mcr.microsoft.com/azureiotedge-agent:1.2.9" ... "image": "mcr.microsoft.com/azureiotedge-hub:1.2.9"You can retrieve a list of all available tags for azureiotedge-hub at https://mcr.microsoft.com/v2/azureiotedge-hub/tags/list
You can retrieve a list of all available tags for azureiotedge-agent at https://mcr.microsoft.com/v2/azureiotedge-agent/tags/list
Thanks!
-
randomrabbit 121 Reputation points
2022-04-28T12:14:24.157+00:00 Thanks for the heads-up! I was using 1.2.7 so I'll be sure to update those. What's the best way to check on the gateway that the updates have been deployed?
-
randomrabbit 121 Reputation points
2022-04-28T12:15:33.05+00:00 Ok, I can see the docker images were updated immediately. Nice. Same warnings remain though.
-
António Sérgio Azevedo 7,661 Reputation points Microsoft Employee
2022-04-28T14:28:10.797+00:00 Thanks for checking this! What happens if you use AMQPS (5671) or HTTPS (443) instead? Can you as well validate that in your leaf device the hostname is resolvable to an IP Address. If not you can try adding a host file entry on the leaf device (in the /etc/hosts file). This is a similar issue to yours: https://github.com/Azure/iotedge/issues/6276
I may need to take some time reproducing this on my side with an IoT Edge using version 1.2.9.
Thank you for your time so far!
-
randomrabbit 121 Reputation points
2022-04-28T15:23:05.77+00:00 From my leaf device, I can run
openssl s_client -connect mygateway.hostname.com:8883 -CAfile <CERTDIR>/certs/azure-iot-test-only.root.ca.cert.pem -showcertsand it works fine. So this makes me think the certificate and the hostname are fine? The issue comes when I try to use the Azure IoT Python SDK:
IoTHubDeviceClient.create_from_connection_string( connection_string, server_verification_cert= open('certfire').read() )so my leaf device is not running IoT Edge. I'll check the other protocols soon. Thanks for your help.
Edit: using a bad hostname in my python code gives a different error, so I think the hostname is resolved ok.
-
randomrabbit 121 Reputation points
2022-04-28T15:39:56.42+00:00 If I open ports 443 and 5671 I can run the openssl s_client command just fine. I don't know how to make the Python SDK use a different protocol.
I should mention I have azure-iot-device version 2.11.0 and certifi version 2021.10.8 on the leaf device.
-
António Sérgio Azevedo 7,661 Reputation points Microsoft Employee
2022-04-29T15:22:41.103+00:00 Hi @randomrabbit it is becoming more clear now that the issue may be related with the Python SDK and not the Protocol being used. It is somehow related with the version of IoT Edge like you refer in the related thread: https://learn.microsoft.com/en-us/answers/questions/828379/index.html
Can you confirm you are using this sample on your leaf device? https://github.com/Azure/azure-iot-sdk-python/blob/main/azure-iot-device/samples/async-edge-scenarios/send_message_downstream.py
Can you try using a different SDK in your Leaf Device? For example Csharp?
Thanks!
-
randomrabbit 121 Reputation points
2022-04-29T15:57:38.473+00:00 Hi. I went back to Edge runtime 1.1 and everything works fine. So looks like the Python SDK does not properly handle the 1.2 runtime for some reason? I see in github there are a couple of tickets open about the python sdk having trouble with certifiates, although in those tickets the exception is not exactly like mine. Maybe you could increase the priority on fixing the python sdk? :)
I don't have time to test other SDKs now. We'll keep Edge runtime 1.1.
-
António Sérgio Azevedo 7,661 Reputation points Microsoft Employee
2022-04-29T16:01:41.327+00:00 Thanks @randomrabbit ,
I am adding to my backlog to reproduce your issue while using Python Leaf device behind IoT Edge Transparent Gateway 1.2 and report back what I found.Thank you so much for all your time so far!!!
Sign in to comment