IoT Edge Ubuntu 18.04 Nvidia Jetson ARM64 - Could not initialize module runtime

Question

I've been trying to install and configure IoT Edge on a custom Ubuntu 18.04 build on an ARM64 architecture for an Nvidia Jetson board. I'm not a kernal/hardware developer but if you have additional questions about the details of that build please let me know what other details I can provide.

From what I can tell, ARM64 and Ubuntu 18.04 should be supported in public preview:
https://learn.microsoft.com/en-us/azure/iot-edge/support#tier-1

I installed IoT Edge by following the instructions here for Ubuntu 18.04: https://learn.microsoft.com/en-us/azure/iot-edge/how-to-install-iot-edge?tabs=linux

I confirmed that the moby requirements script appeared to pass the requirements described in the documentation (general requirements and network ones).

I also created test certificates as described here: https://learn.microsoft.com/en-us/azure/iot-edge/how-to-create-test-certificates#linux-1

Then uploaded and verified the certificate in DPS, connected it to the IoT Hub, and created an enrollment group with the certificate for IoT Edge devices.

After that, I added config.yaml file as described to test the DPS X509 group enrollment provisioning process:

   provisioning:  
     source: "dps"  
     global_endpoint: "https://global.azure-devices-provisioning.net"  
     scope_id: "<OMITTED>"  
     attestation:  
       method: "x509"  
       identity_cert: "<OMITTED>"  
       identity_pk: "<OMITTED>"  
   agent:  
     name: "edgeAgent"  
     type: "docker"  
     env: {}  
     config:  
       image: "mcr.microsoft.com/azureiotedge-agent:1.0"  
       auth: {}  
   hostname: "<OMITTED>"  
   connect:  
     management_uri: "unix:///var/run/iotedge/mgmt.sock"  
     workload_uri: "unix:///var/run/iotedge/workload.sock"  
   listen:  
     management_uri: "fd://iotedge.mgmt.socket"  
     workload_uri: "fd://iotedge.socket"  
   homedir: "/var/lib/iotedge"  
   moby_runtime:  
     uri: "unix:///var/run/docker.sock"  

And then I tried to start the IoT Edge daemon with:

   sudo systemctl restart iotedge  

I was hoping that this would finish things up but here is where the errors started. Initially, I had an issue with my file:// prefix but then I
set the location of the certificates and the scope ID correctly and made sure to use the file:// prefix before the /path/to/certs.

Here are some of the logs I'm currently getting from IoT Edge:

   Oct 19 17:24:48 jetson systemd[1]: Started Azure IoT Edge daemon.  
   Oct 19 17:24:48 jetson iotedged[21128]: 2020-10-19T21:24:48Z [INFO] - Starting Azure IoT Edge Security Daemon  
   Oct 19 17:24:48 jetson iotedged[21128]: 2020-10-19T21:24:48Z [INFO] - Version - 1.0.10  
   Oct 19 17:24:48 jetson iotedged[21128]: 2020-10-19T21:24:48Z [INFO] - Using config file: /etc/iotedge/config.yaml  
   Oct 19 17:24:48 jetson iotedged[21128]: 2020-10-19T21:24:48Z [INFO] - Configuring /var/lib/iotedge as the home directory.  
   Oct 19 17:24:48 jetson iotedged[21128]: 2020-10-19T21:24:48Z [INFO] - Configuring certificates...  
   Oct 19 17:24:48 jetson iotedged[21128]: 2020-10-19T21:24:48Z [INFO] - Transparent gateway certificates not found, operating in quick start mode...  
   Oct 19 17:24:48 jetson iotedged[21128]: 2020-10-19T21:24:48Z [INFO] - Finished configuring provisioning environment variables and certificates.  
   Oct 19 17:24:48 jetson iotedged[21128]: 2020-10-19T21:24:48Z [INFO] - Initializing hsm...  
   Oct 19 17:24:48 jetson iotedged[21128]: 2020-10-19T21:24:48Z [INFO] - Finished initializing hsm.  
   Oct 19 17:24:48 jetson iotedged[21128]: 2020-10-19T21:24:48Z [INFO] - Initializing hsm X509 interface...  
   Oct 19 17:24:48 jetson iotedged[21128]: 2020-10-19T21:24:48Z [INFO] - Finished initializing hsm X509 interface...  
   Oct 19 17:24:48 jetson iotedged[21128]: 2020-10-19T21:24:48Z [INFO] - Provisioning edge device...  
   Oct 19 17:24:48 jetson iotedged[21128]: 2020-10-19T21:24:48Z [INFO] - Error loading the hybrid identity key. Re-creating a new key. The daemon could not start up successfully: Could not load the hybrid identity key.  
   Oct 19 17:24:48 jetson iotedged[21128]: 2020-10-19T21:24:48Z [INFO] - Starting provisioning edge device via X509 provisioning...  
   Oct 19 17:24:48 jetson iotedged[21128]: 2020-10-19T21:24:48Z [INFO] - Starting DPS registration with scope_id "<SCOPE_ID_OMITTED>", registration_id "iot-edge-test-fmc"  
   Oct 19 17:24:58 jetson iotedged[21128]: 2020-10-19T21:24:58Z [INFO] - DPS registration assigned device "iot-edge-test-fmc" in hub "IoT-Edge-Hub-Test.azure-devices.net"  
   Oct 19 17:24:58 jetson iotedged[21128]: 2020-10-19T21:24:58Z [INFO] - Successful DPS provisioning.  
   Oct 19 17:24:58 jetson iotedged[21128]: 2020-10-19T21:24:58Z [INFO] - Finished provisioning edge device.  
   Oct 19 17:24:58 jetson iotedged[21128]: 2020-10-19T21:24:58Z [INFO] - Initializing the module runtime...  
   Oct 19 17:24:58 jetson iotedged[21128]: 2020-10-19T21:24:58Z [INFO] - Initializing module runtime...  
   Oct 19 17:24:58 jetson iotedged[21128]: 2020-10-19T21:24:58Z [INFO] - Using runtime network id azure-iot-edge  
   Oct 19 17:24:58 jetson iotedged[21128]: 2020-10-19T21:24:58Z [WARN] - Could not initialize module runtime  
   Oct 19 17:24:58 jetson iotedged[21128]: 2020-10-19T21:24:58Z [WARN] -         caused by: Container runtime error  
   Oct 19 17:24:58 jetson iotedged[21128]: 2020-10-19T21:24:58Z [WARN] -         caused by: error trying to connect: Connection refused (os error 111)  
   Oct 19 17:24:58 jetson iotedged[21128]: 2020-10-19T21:24:58Z [ERR!] - The daemon could not start up successfully: Could not initialize module runtime  
   Oct 19 17:24:58 jetson iotedged[21128]: 2020-10-19T21:24:58Z [ERR!] -         caused by: Could not initialize module runtime  
   Oct 19 17:24:58 jetson iotedged[21128]: 2020-10-19T21:24:58Z [ERR!] -         caused by: Container runtime error  
   Oct 19 17:24:58 jetson iotedged[21128]: 2020-10-19T21:24:58Z [ERR!] -         caused by: error trying to connect: Connection refused (os error 111)  
   Oct 19 17:24:58 jetson iotedged[21128]: 2020-10-19T21:24:48Z [INFO] (/project/edgelet/hsm-sys/azure-iot-hsm-c/src/hsm_log.c:log_init:41) Initialized logging  
   Oct 19 17:24:58 jetson systemd[1]: iotedge.service: Main process exited, code=exited, status=1/FAILURE  
   Oct 19 17:24:58 jetson systemd[1]: iotedge.service: Failed with result 'exit-code'.  
   Oct 19 17:24:59 jetson systemd[1]: iotedge.service: Service hold-off time over, scheduling restart.  
   Oct 19 17:24:59 jetson systemd[1]: iotedge.service: Scheduled restart job, restart counter is at 1.  
   Oct 19 17:24:59 jetson systemd[1]: Stopped Azure IoT Edge daemon.  
   Oct 19 17:24:59 jetson systemd[1]: Started Azure IoT Edge daemon.  
   Oct 19 17:24:59 jetson iotedged[21308]: 2020-10-19T21:24:59Z [INFO] - Starting Azure IoT Edge Security Daemon  
   Oct 19 17:24:59 jetson iotedged[21308]: 2020-10-19T21:24:59Z [INFO] - Version - 1.0.10  
   Oct 19 17:24:59 jetson iotedged[21308]: 2020-10-19T21:24:59Z [INFO] - Using config file: /etc/iotedge/config.yaml  
   Oct 19 17:24:59 jetson iotedged[21308]: 2020-10-19T21:24:59Z [INFO] - Configuring /var/lib/iotedge as the home directory.  
   Oct 19 17:24:59 jetson iotedged[21308]: 2020-10-19T21:24:59Z [INFO] - Configuring certificates...  
   Oct 19 17:24:59 jetson iotedged[21308]: 2020-10-19T21:24:59Z [INFO] - Transparent gateway certificates not found, operating in quick start mode...  
   Oct 19 17:24:59 jetson iotedged[21308]: 2020-10-19T21:24:59Z [INFO] - Finished configuring provisioning environment variables and certificates.  
   Oct 19 17:24:59 jetson iotedged[21308]: 2020-10-19T21:24:59Z [INFO] - Initializing hsm...  
   Oct 19 17:24:59 jetson iotedged[21308]: 2020-10-19T21:24:59Z [INFO] - Finished initializing hsm.  
   Oct 19 17:24:59 jetson iotedged[21308]: 2020-10-19T21:24:59Z [INFO] - Initializing hsm X509 interface...  
   Oct 19 17:24:59 jetson iotedged[21308]: 2020-10-19T21:24:59Z [INFO] - Finished initializing hsm X509 interface...  
   Oct 19 17:24:59 jetson iotedged[21308]: 2020-10-19T21:24:59Z [INFO] - Provisioning edge device...  
   Oct 19 17:24:59 jetson iotedged[21308]: 2020-10-19T21:24:59Z [INFO] - Starting provisioning edge device via X509 provisioning...  
   Oct 19 17:24:59 jetson iotedged[21308]: 2020-10-19T21:24:59Z [INFO] - Starting DPS registration with scope_id "<SCOPE_ID_OMITTED>", registration_id "iot-edge-test-fmc"  
   Oct 19 17:25:09 jetson iotedged[21308]: 2020-10-19T21:25:09Z [INFO] - DPS registration assigned device "iot-edge-test-fmc" in hub "IoT-Edge-Hub-Test.azure-devices.net"  
   Oct 19 17:25:09 jetson iotedged[21308]: 2020-10-19T21:25:09Z [INFO] - Successful DPS provisioning.  
   Oct 19 17:25:09 jetson iotedged[21308]: 2020-10-19T21:25:09Z [INFO] - Finished provisioning edge device.  
   Oct 19 17:25:09 jetson iotedged[21308]: 2020-10-19T21:25:09Z [INFO] - Initializing the module runtime...  
   Oct 19 17:25:09 jetson iotedged[21308]: 2020-10-19T21:25:09Z [INFO] - Initializing module runtime...  
   Oct 19 17:25:09 jetson iotedged[21308]: 2020-10-19T21:25:09Z [INFO] - Using runtime network id azure-iot-edge  
   Oct 19 17:25:09 jetson iotedged[21308]: 2020-10-19T21:25:09Z [WARN] - Could not initialize module runtime  
   Oct 19 17:25:09 jetson iotedged[21308]: 2020-10-19T21:25:09Z [WARN] -         caused by: Container runtime error  
   Oct 19 17:25:09 jetson iotedged[21308]: 2020-10-19T21:25:09Z [WARN] -         caused by: error trying to connect: Connection refused (os error 111)  
   Oct 19 17:25:09 jetson iotedged[21308]: 2020-10-19T21:25:09Z [ERR!] - The daemon could not start up successfully: Could not initialize module runtime  
   Oct 19 17:25:09 jetson iotedged[21308]: 2020-10-19T21:25:09Z [ERR!] -         caused by: Could not initialize module runtime  
   Oct 19 17:25:09 jetson iotedged[21308]: 2020-10-19T21:25:09Z [ERR!] -         caused by: Container runtime error  
   Oct 19 17:25:09 jetson iotedged[21308]: 2020-10-19T21:25:09Z [ERR!] -         caused by: error trying to connect: Connection refused (os error 111)  
   Oct 19 17:25:09 jetson iotedged[21308]: 2020-10-19T21:24:59Z [INFO] (/project/edgelet/hsm-sys/azure-iot-hsm-c/src/hsm_log.c:log_init:41) Initialized logging  
   Oct 19 17:25:09 jetson systemd[1]: iotedge.service: Main process exited, code=exited, status=1/FAILURE  
   Oct 19 17:25:09 jetson systemd[1]: iotedge.service: Failed with result 'exit-code'.  
   Oct 19 17:25:10 jetson systemd[1]: iotedge.service: Service hold-off time over, scheduling restart.  
   .......   
   .......  
   A lot more restarts  

The device seems to be connected to DPS because I see it registering a new IoT Edge device in the Azure Portal, but the status on that device has an N/A runtime response and a module count of 0 across the board (not even the edge agent

I thought maybe the issues were related to a test cert that I had reused from a personal test device I'd already connected to IoT Hub so I reprovisioned all the certificates, created a new cert/enrollment group, and added the certs to the device again in the proper paths but still no luck.

On top of all this, apparently this happened during some downtime for the Azure portal so not entirely sure if it's possible something that went weird then is cascading into now.

Any ideas for what to debug next?

Answer 1

@QuantumCache We're in luck actually! We looked into this more and it was an issue with the docker service not starting because of a change we made in the build. Thanks for your help!

Answer 2

Hello @QuantumCache ! Thanks for following up. Here are answers to your questions, please let me know if I can provide additional logs to help debug this.

Just wanted to know which environment is this? POC/Test?
Right now we're migrating to a prototype board. Do you mean some specific environment detail in the context of Azure? I might need more clarification here.

Did you try this device provisioning after some point of time-keeping in mind the Azure portal downtime issue?
I waited for a bit and tried provisioning again after the issues appear to have finished. We also tried again today but no luck. The same issues shown above.

Did you try re-creating the test certs fresh and used for provisioning?
Yes, I tried fresh certs. The device actually appears in the IoT Hub after running sudo systemctl restart iotedge and goes through the provisioning process. But it never reports back the status and the edgeAgenet and edgeHub don't spin up on the device and aren't reported as running in the console.

Any other IoT devices have you tried?
I've finished the setup process for a Raspberry Pi just to test the DPS part of the provisioning, that was working fine. I think the issue is related to docker (moby) not starting or not working correctly with iotedge on the board.

What was the Azure Portal Issue which you have encountered? is it related to IoT or to the general Azure Portal like 'Loading issue'/'Login issue' ?
The IoT Hub portal wasn't loading devices or deployment configurations and was having long latencies. Then later it wasn't letting me in at all and I searched and it appeared there were login issues yesterday - so yes I think it was related to the loading/login issues yesterday. But I don't know if those issues were related to the real issue necessarily.