Provisioning to IOT Hub on Jetpack 5.0.2

Cabrejos, Jason J 10 Reputation points
2023-08-16T19:15:32.65+00:00

I have an nvidia jetson on Jetpack 5.0.2 that ive been trying to provision on iothub for while. Previously I have provisioned jetsons on Jetpack 4.6 with no issues, however 5.0.2 has posed problematic. I am looking for help to try to address this issue. This is a newly flashed Jetson with 5.0.2. I believe something on Jetpack 5.0.2 is blocking certificate from being read but I am not familiar enough with iothub to debug the issue. I have read other forum posts and they have not cleared my issue. but I am willing to try again and post the results on here

Output from iotedge check

sudo iotedge check --verbose

Configuration checks (aziot-identity-service)
---------------------------------------------
√ keyd configuration is well-formed - OK
√ certd configuration is well-formed - OK
√ tpmd configuration is well-formed - OK
√ identityd configuration is well-formed - OK
√ daemon configurations up-to-date with config.toml - OK
√ identityd config toml file specifies a valid hostname - OK
√ aziot-identity-service package is up-to-date - OK
√ host time is close to reference time - OK
√ preloaded certificates are valid - OK
√ keyd is running - OK
√ certd is running - OK
√ identityd is running - OK
× read all preloaded certificates from the Certificates Service - Error
    could not load cert with ID "aziot-edged-trust-bundle"
    
    Caused by:
        parameter "id" has an invalid value
        caused by: not found
        caused by: could not load cert with ID "aziot-edged-trust-bundle"
                   
                   Caused by:
                       parameter "id" has an invalid value
                       caused by: not found
√ read all preloaded key pairs from the Keys Service - OK
√ check all EST server URLs utilize HTTPS - OK
√ ensure all preloaded certificates match preloaded private keys with the same ID - OK

Connectivity checks (aziot-identity-service)
--------------------------------------------
‼ host can connect to and perform TLS handshake with iothub AMQP port - Warning
    Could not retrieve iothub_hostname from provisioning file.
    Please specify the backing IoT Hub name using --iothub-hostname switch if you have that information.
    Since no hostname is provided, all hub connectivity tests will be skipped.
        caused by: Could not retrieve iothub_hostname from provisioning file.
                   Please specify the backing IoT Hub name using --iothub-hostname switch if you have that information.
                   Since no hostname is provided, all hub connectivity tests will be skipped.
‼ host can connect to and perform TLS handshake with iothub HTTPS / WebSockets port - Warning
    Could not retrieve iothub_hostname from provisioning file.
    Please specify the backing IoT Hub name using --iothub-hostname switch if you have that information.
    Since no hostname is provided, all hub connectivity tests will be skipped.
        caused by: Could not retrieve iothub_hostname from provisioning file.
                   Please specify the backing IoT Hub name using --iothub-hostname switch if you have that information.
                   Since no hostname is provided, all hub connectivity tests will be skipped.
‼ host can connect to and perform TLS handshake with iothub MQTT port - Warning
    Could not retrieve iothub_hostname from provisioning file.
    Please specify the backing IoT Hub name using --iothub-hostname switch if you have that information.
    Since no hostname is provided, all hub connectivity tests will be skipped.
        caused by: Could not retrieve iothub_hostname from provisioning file.
                   Please specify the backing IoT Hub name using --iothub-hostname switch if you have that information.
                   Since no hostname is provided, all hub connectivity tests will be skipped.
√ host can connect to and perform TLS handshake with DPS endpoint - OK

Configuration checks
--------------------
√ aziot-edged configuration is well-formed - OK
√ configuration up-to-date with config.toml - OK
√ container engine is installed and functional - OK
× configuration has correct URIs for daemon mgmt endpoint - Error
    SocketError - SocketErrorCode (TimedOut) : Operation timed out
    One or more errors occurred. (Got bad response: )
        caused by: SocketError - SocketErrorCode (TimedOut) : Operation timed out
                   One or more errors occurred. (Got bad response: )
        caused by: docker returned exit status: 1, stderr = SocketError - SocketErrorCode (TimedOut) : Operation timed out
                   One or more errors occurred. (Got bad response: )
√ aziot-edge package is up-to-date - OK
√ container time is close to host time - OK
‼ DNS server - Warning
    Container engine is not configured with DNS server setting, which may impact connectivity to IoT Hub.
    Please see https://aka.ms/iotedge-prod-checklist-dns for best practices.
    You can ignore this warning if you are setting DNS server per module in the Edge deployment.
        caused by: Container engine is not configured with DNS server setting, which may impact connectivity to IoT Hub.
                   Please see https://aka.ms/iotedge-prod-checklist-dns for best practices.
                   You can ignore this warning if you are setting DNS server per module in the Edge deployment.
‼ production readiness: logs policy - Warning
    Container engine is not configured to rotate module logs which may cause it run out of disk space.
    Please see https://aka.ms/iotedge-prod-checklist-logs for best practices.
    You can ignore this warning if you are setting log policy per module in the Edge deployment.
        caused by: Container engine is not configured to rotate module logs which may cause it run out of disk space.
                   Please see https://aka.ms/iotedge-prod-checklist-logs for best practices.
                   You can ignore this warning if you are setting log policy per module in the Edge deployment.
‼ production readiness: Edge Agent's storage directory is persisted on the host filesystem - Warning
    The edgeAgent module is not configured to persist its /tmp/edgeAgent directory on the host filesystem.
    Data might be lost if the module is deleted or updated.
    Please see https://aka.ms/iotedge-storage-host for best practices.
        caused by: The edgeAgent module is not configured to persist its /tmp/edgeAgent directory on the host filesystem.
                   Data might be lost if the module is deleted or updated.
                   Please see https://aka.ms/iotedge-storage-host for best practices.
‼ production readiness: Edge Hub's storage directory is persisted on the host filesystem - Warning
    The edgeHub module is not configured to persist its /tmp/edgeHub directory on the host filesystem.
    Data might be lost if the module is deleted or updated.
    Please see https://aka.ms/iotedge-storage-host for best practices.
        caused by: The edgeHub module is not configured to persist its /tmp/edgeHub directory on the host filesystem.
                   Data might be lost if the module is deleted or updated.
                   Please see https://aka.ms/iotedge-storage-host for best practices.
‼ Agent image is valid and can be pulled from upstream - Warning
    skipping because of previous failures
√ proxy settings are consistent in aziot-edged, aziot-identityd, moby daemon and config.toml - OK

Connectivity checks
-------------------
‼ container on the default network can connect to upstream AMQP port - Warning
    skipping because of previous failures
‼ container on the default network can connect to upstream HTTPS / WebSockets port - Warning
    skipping because of previous failures
‼ container on the default network can connect to upstream MQTT port - Warning
    skipping because of previous failures
‼ container on the IoT Edge module network can connect to upstream AMQP port - Warning
    skipping because of previous failures
‼ container on the IoT Edge module network can connect to upstream HTTPS / WebSockets port - Warning
    skipping because of previous failures
‼ container on the IoT Edge module network can connect to upstream MQTT port - Warning
    skipping because of previous failures
22 check(s) succeeded.
7 check(s) raised warnings.
2 check(s) raised errors.
7 check(s) were skipped due to errors from other checks.

Output from iotedge system logs

Aug 16 13:56:13 nvidia-desktop aziot-identityd[45487]: 2023-08-16T18:56:13Z [INFO] - Starting service...
Aug 16 13:56:13 nvidia-desktop aziot-identityd[45487]: 2023-08-16T18:56:13Z [INFO] - Version - 1.4.5
Aug 16 13:56:13 nvidia-desktop aziot-identityd[45487]: 2023-08-16T18:56:13Z [INFO] - Provisioning starting. Reason: Startup
Aug 16 13:56:13 nvidia-desktop aziot-keyd[43423]: 2023-08-16T18:56:13Z [INFO] - <-- GET /key/device-id?api-version=2021-05-01 {"host": "keyd.sock"}
Aug 16 13:56:13 nvidia-desktop aziot-keyd[43423]: 2023-08-16T18:56:13Z [INFO] - --> 200 {"content-type": "application/json"}
Aug 16 13:56:13 nvidia-desktop aziot-keyd[43423]: 2023-08-16T18:56:13Z [INFO] - <-- POST /sign?api-version=2021-05-01 {"content-type": "application/json", "host": "keyd.sock", "content-length": "414"}
Aug 16 13:56:13 nvidia-desktop aziot-keyd[43423]: 2023-08-16T18:56:13Z [INFO] - --> 200 {"content-type": "application/json"}
Aug 16 13:56:13 nvidia-desktop aziot-identityd[45487]: 2023-08-16T18:56:13Z [INFO] - Sending DPS registration request.
Aug 16 13:56:13 nvidia-desktop aziot-identityd[45487]: 2023-08-16T18:56:13Z [ERR!] - Failed to provision with IoT Hub, and no valid device backup was found: DPS client error
Aug 16 13:56:13 nvidia-desktop aziot-identityd[45487]: 2023-08-16T18:56:13Z [ERR!] - service encountered an error
Aug 16 13:56:13 nvidia-desktop aziot-identityd[45487]: 2023-08-16T18:56:13Z [ERR!] - caused by: DPS client error
Aug 16 13:56:13 nvidia-desktop aziot-identityd[45487]: 2023-08-16T18:56:13Z [ERR!] - caused by: Unauthorized
Aug 16 13:56:13 nvidia-desktop aziot-identityd[45487]: 2023-08-16T18:56:13Z [ERR!] -    0: <unknown>
Aug 16 13:56:13 nvidia-desktop aziot-identityd[45487]:    1: <unknown>
Aug 16 13:56:13 nvidia-desktop systemd[1]: aziot-identityd.service: Main process exited, code=exited, status=1/FAILURE
Aug 16 13:56:13 nvidia-desktop systemd[1]: aziot-identityd.service: Failed with result 'exit-code'.
Aug 16 13:56:18 nvidia-desktop aziot-edged[43388]: 2023-08-16T18:56:18Z [INFO] - Obtaining Edge device provisioning data...
Aug 16 13:56:18 nvidia-desktop systemd[1]: aziot-identityd.service: Scheduled restart job, restart counter is at 67.
Aug 16 13:56:18 nvidia-desktop systemd[1]: Stopped Azure IoT Identity Service.
Aug 16 13:56:18 nvidia-desktop aziot-edged[43388]: 2023-08-16T18:56:18Z [WARN] - Failed to send HTTP request (attempt 1 of 2): connection error: Connection reset by peer (os error 104)
Aug 16 13:56:18 nvidia-desktop systemd[1]: Started Azure IoT Identity Service.
Aug 16 13:56:18 nvidia-desktop aziot-identityd[45498]: 2023-08-16T18:56:18Z [INFO] - Starting service...
Aug 16 13:56:18 nvidia-desktop aziot-identityd[45498]: 2023-08-16T18:56:18Z [INFO] - Version - 1.4.5
Aug 16 13:56:18 nvidia-desktop aziot-identityd[45498]: 2023-08-16T18:56:18Z [INFO] - Provisioning starting. Reason: Startup
Aug 16 13:56:18 nvidia-desktop aziot-keyd[43423]: 2023-08-16T18:56:18Z [INFO] - <-- GET /key/device-id?api-version=2021-05-01 {"host": "keyd.sock"}
Aug 16 13:56:18 nvidia-desktop aziot-keyd[43423]: 2023-08-16T18:56:18Z [INFO] - --> 200 {"content-type": "application/json"}
Aug 16 13:56:18 nvidia-desktop aziot-keyd[43423]: 2023-08-16T18:56:18Z [INFO] - <-- POST /sign?api-version=2021-05-01 {"content-type": "application/json", "host": "keyd.sock", "content-length": "414"}
Aug 16 13:56:18 nvidia-desktop aziot-keyd[43423]: 2023-08-16T18:56:18Z [INFO] - --> 200 {"content-type": "application/json"}
Aug 16 13:56:18 nvidia-desktop aziot-identityd[45498]: 2023-08-16T18:56:18Z [INFO] - Sending DPS registration request.
Aug 16 13:56:19 nvidia-desktop aziot-identityd[45498]: 2023-08-16T18:56:19Z [ERR!] - Failed to provision with IoT Hub, and no valid device backup was found: DPS client error
Aug 16 13:56:19 nvidia-desktop aziot-identityd[45498]: 2023-08-16T18:56:19Z [ERR!] - service encountered an error
Aug 16 13:56:19 nvidia-desktop aziot-identityd[45498]: 2023-08-16T18:56:19Z [ERR!] - caused by: DPS client error
Aug 16 13:56:19 nvidia-desktop aziot-identityd[45498]: 2023-08-16T18:56:19Z [ERR!] - caused by: Unauthorized
Aug 16 13:56:19 nvidia-desktop aziot-identityd[45498]: 2023-08-16T18:56:19Z [ERR!] -    0: <unknown>
Aug 16 13:56:19 nvidia-desktop aziot-identityd[45498]:    1: <unknown>
Aug 16 13:56:19 nvidia-desktop systemd[1]: aziot-identityd.service: Main process exited, code=exited, status=1/FAILURE
Aug 16 13:56:19 nvidia-desktop systemd[1]: aziot-identityd.service: Failed with result 'exit-code'.
Aug 16 13:56:22 nvidia-desktop aziot-keyd[43423]: 2023-08-16T18:56:22Z [INFO] - <-- GET /key/defender_encryption_key?api-version=2020-09-01 {"host": "foo", "user-agent": "libcurl-agent/1.0", "accept": "*/*"}
Aug 16 13:56:22 nvidia-desktop aziot-keyd[43423]: 2023-08-16T18:56:22Z [INFO] - --> 200 {"content-type": "application/json"}
Aug 16 13:56:24 nvidia-desktop systemd[1]: aziot-identityd.service: Scheduled restart job, restart counter is at 68.
Aug 16 13:56:24 nvidia-desktop systemd[1]: Stopped Azure IoT Identity Service.
Aug 16 13:56:24 nvidia-desktop aziot-edged[43388]: 2023-08-16T18:56:24Z [WARN] - Failed to send HTTP request (attempt 2 of 2): connection error: Connection reset by peer (os error 104)
Aug 16 13:56:24 nvidia-desktop aziot-edged[43388]: 2023-08-16T18:56:24Z [ERR!] - Failed to obtain device identity: connection error: Connection reset by peer (os error 104)
Aug 16 13:56:24 nvidia-desktop aziot-edged[43388]: 2023-08-16T18:56:24Z [INFO] - Requesting device reprovision
Aug 16 13:56:24 nvidia-desktop aziot-edged[43388]: 2023-08-16T18:56:24Z [WARN] - Failed to clear provisioning cache before reprovision: No such file or directory (os error 2)
Aug 16 13:56:24 nvidia-desktop aziot-edged[43388]: 2023-08-16T18:56:24Z [WARN] - Failed to send HTTP request (attempt 1 of 2): error trying to connect: Connection refused (os error 111)
Aug 16 13:56:24 nvidia-desktop systemd[1]: Started Azure IoT Identity Service.
Aug 16 13:56:24 nvidia-desktop aziot-identityd[45515]: 2023-08-16T18:56:24Z [INFO] - Starting service...
Aug 16 13:56:24 nvidia-desktop aziot-identityd[45515]: 2023-08-16T18:56:24Z [INFO] - Version - 1.4.5
Aug 16 13:56:24 nvidia-desktop aziot-identityd[45515]: 2023-08-16T18:56:24Z [INFO] - Provisioning starting. Reason: Startup
Aug 16 13:56:24 nvidia-desktop aziot-keyd[43423]: 2023-08-16T18:56:24Z [INFO] - <-- GET /key/device-id?api-version=2021-05-01 {"host": "keyd.sock"}
Aug 16 13:56:24 nvidia-desktop aziot-keyd[43423]: 2023-08-16T18:56:24Z [INFO] - --> 200 {"content-type": "application/json"}
Aug 16 13:56:24 nvidia-desktop aziot-keyd[43423]: 2023-08-16T18:56:24Z [INFO] - <-- POST /sign?api-version=2021-05-01 {"content-type": "application/json", "host": "keyd.sock", "content-length": "414"}
Aug 16 13:56:24 nvidia-desktop aziot-keyd[43423]: 2023-08-16T18:56:24Z [INFO] - --> 200 {"content-type": "application/json"}
Aug 16 13:56:24 nvidia-desktop aziot-identityd[45515]: 2023-08-16T18:56:24Z [INFO] - Sending DPS registration request.
Aug 16 13:56:24 nvidia-desktop aziot-identityd[45515]: 2023-08-16T18:56:24Z [ERR!] - Failed to provision with IoT Hub, and no valid device backup was found: DPS client error
Aug 16 13:56:24 nvidia-desktop aziot-identityd[45515]: 2023-08-16T18:56:24Z [ERR!] - service encountered an error
Aug 16 13:56:24 nvidia-desktop aziot-identityd[45515]: 2023-08-16T18:56:24Z [ERR!] - caused by: DPS client error
Aug 16 13:56:24 nvidia-desktop aziot-identityd[45515]: 2023-08-16T18:56:24Z [ERR!] - caused by: Unauthorized
Aug 16 13:56:24 nvidia-desktop aziot-identityd[45515]: 2023-08-16T18:56:24Z [ERR!] -    0: <unknown>
Aug 16 13:56:24 nvidia-desktop aziot-identityd[45515]:    1: <unknown>
Aug 16 13:56:24 nvidia-desktop systemd[1]: aziot-identityd.service: Main process exited, code=exited, status=1/FAILURE
Aug 16 13:56:24 nvidia-desktop systemd[1]: aziot-identityd.service: Failed with result 'exit-code'.

Output from ./check-config.sh as designated from the provisioning tutorial

./check-config.sh 
info: reading kernel config from /proc/config.gz ...

Generally Necessary:
- cgroup hierarchy: properly mounted [/sys/fs/cgroup]
- CONFIG_NAMESPACES: enabled
- CONFIG_NET_NS: enabled
- CONFIG_PID_NS: enabled
- CONFIG_IPC_NS: enabled
- CONFIG_UTS_NS: enabled
- CONFIG_CGROUPS: enabled
- CONFIG_CGROUP_CPUACCT: enabled
- CONFIG_CGROUP_DEVICE: enabled
- CONFIG_CGROUP_FREEZER: enabled
- CONFIG_CGROUP_SCHED: enabled
- CONFIG_CPUSETS: enabled
- CONFIG_MEMCG: enabled
- CONFIG_KEYS: enabled
- CONFIG_VETH: enabled (as module)
- CONFIG_BRIDGE: enabled
- CONFIG_BRIDGE_NETFILTER: enabled (as module)
- CONFIG_IP_NF_FILTER: enabled (as module)
- CONFIG_IP_NF_TARGET_MASQUERADE: enabled (as module)
- CONFIG_NETFILTER_XT_MATCH_ADDRTYPE: enabled (as module)
- CONFIG_NETFILTER_XT_MATCH_CONNTRACK: enabled (as module)
- CONFIG_NETFILTER_XT_MATCH_IPVS: enabled (as module)
- CONFIG_NETFILTER_XT_MARK: enabled (as module)
- CONFIG_IP_NF_NAT: enabled (as module)
- CONFIG_NF_NAT: enabled (as module)
- CONFIG_POSIX_MQUEUE: enabled
- CONFIG_CGROUP_BPF: missing

Optional Features:
- CONFIG_USER_NS: enabled
- CONFIG_SECCOMP: enabled
- CONFIG_SECCOMP_FILTER: enabled
- CONFIG_CGROUP_PIDS: enabled
- CONFIG_MEMCG_SWAP: enabled
    (cgroup swap accounting is currently enabled)
- CONFIG_BLK_CGROUP: enabled
- CONFIG_BLK_DEV_THROTTLING: enabled
- CONFIG_CGROUP_PERF: enabled
- CONFIG_CGROUP_HUGETLB: enabled
- CONFIG_NET_CLS_CGROUP: enabled
- CONFIG_CGROUP_NET_PRIO: enabled
- CONFIG_CFS_BANDWIDTH: enabled
- CONFIG_FAIR_GROUP_SCHED: enabled
- CONFIG_RT_GROUP_SCHED: enabled
- CONFIG_IP_NF_TARGET_REDIRECT: enabled (as module)
- CONFIG_IP_VS: enabled (as module)
- CONFIG_IP_VS_NFCT: enabled
- CONFIG_IP_VS_PROTO_TCP: enabled
- CONFIG_IP_VS_PROTO_UDP: enabled
- CONFIG_IP_VS_RR: enabled (as module)
- CONFIG_SECURITY_SELINUX: enabled
- CONFIG_SECURITY_APPARMOR: missing
- CONFIG_EXT4_FS: enabled
- CONFIG_EXT4_FS_POSIX_ACL: enabled
- CONFIG_EXT4_FS_SECURITY: enabled
- Network Drivers:
  - "overlay":
    - CONFIG_VXLAN: enabled
    - CONFIG_BRIDGE_VLAN_FILTERING: enabled
      Optional (for encrypted networks):
      - CONFIG_CRYPTO: enabled
      - CONFIG_CRYPTO_AEAD: enabled
      - CONFIG_CRYPTO_GCM: enabled (as module)
      - CONFIG_CRYPTO_SEQIV: enabled
      - CONFIG_CRYPTO_GHASH: enabled
      - CONFIG_XFRM: enabled
      - CONFIG_XFRM_USER: enabled
      - CONFIG_XFRM_ALGO: enabled
      - CONFIG_INET_ESP: enabled (as module)
  - "ipvlan":
    - CONFIG_IPVLAN: enabled (as module)
  - "macvlan":
    - CONFIG_MACVLAN: enabled (as module)
    - CONFIG_DUMMY: enabled
  - "ftp,tftp client in container":
    - CONFIG_NF_NAT_FTP: enabled (as module)
    - CONFIG_NF_CONNTRACK_FTP: enabled (as module)
    - CONFIG_NF_NAT_TFTP: enabled (as module)
    - CONFIG_NF_CONNTRACK_TFTP: enabled (as module)
- Storage Drivers:
  - "aufs":
    - CONFIG_AUFS_FS: missing
  - "btrfs":
    - CONFIG_BTRFS_FS: enabled (as module)
    - CONFIG_BTRFS_FS_POSIX_ACL: enabled
  - "devicemapper":
    - CONFIG_BLK_DEV_DM: enabled
    - CONFIG_DM_THIN_PROVISIONING: missing
  - "overlay":
    - CONFIG_OVERLAY_FS: enabled (as module)
  - "zfs":
    - /dev/zfs: missing
    - zfs command: missing
    - zpool command: missing

Limits:
- /proc/sys/kernel/keys/root_maxkeys: 1000000

Thank you for your time :)

Azure IoT Hub
Azure IoT Hub
An Azure service that enables bidirectional communication between internet of things (IoT) devices and applications.
1,273 questions
{count} vote

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.