Tutorial: Create a hierarchy of IoT Edge devices
Applies to: IoT Edge 1.4
Deploy Azure IoT Edge nodes across networks organized in hierarchical layers. Each layer in a hierarchy is a gateway device that handles messages and requests from devices in the layer beneath it. This setup is also known as "nested edge".
You can structure a hierarchy of devices so that only the top layer has connectivity to the cloud, and the lower layers can only communicate with adjacent north and south layers. This network layering is the foundation of most industrial networks, which follow the ISA-95 standard.
The goal of this tutorial is to create a hierarchy of IoT Edge devices that simulates a simplified production environment. At the end, you will deploy the Simulated Temperature Sensor module to a lower layer device without internet access by downloading container images through the hierarchy.
To accomplish this goal, this tutorial walks you through creating a hierarchy of IoT Edge devices, deploying IoT Edge runtime containers to your devices, and configuring your devices locally. In this tutorial, you use an automated configuration tool to:
- Create and define the relationships in a hierarchy of IoT Edge devices.
- Configure the IoT Edge runtime on the devices in your hierarchy.
- Install consistent certificates across your device hierarchy.
- Add workloads to the devices in your hierarchy.
- Use the IoT Edge API Proxy module to securely route HTTP traffic over a single port from your lower layer devices.
This tutorial includes a mixture of manual and automated steps to provide a showcase of nested IoT Edge features.
If you would like an entirely automated look at setting up a hierarchy of IoT Edge devices, you can follow the scripted Azure IoT Edge for Industrial IoT sample. This scripted scenario deploys Azure virtual machines as preconfigured devices to simulate a factory environment.
If you would like an in-depth look at the manual steps to create and manage a hierarchy of IoT Edge devices, see the how-to guide on IoT Edge device gateway hierarchies.
In this tutorial, the following network layers are defined:
Top layer: IoT Edge devices at this layer can connect directly to the cloud.
Lower layers: IoT Edge devices at layers below the top layer cannot connect directly to the cloud. They need to go through one or more intermediary IoT Edge devices to send and receive data.
This tutorial uses a two device hierarchy for simplicity, pictured below. One device, the top layer device, represents a device at the top layer of the hierarchy, which can connect directly to the cloud. This device will also be referred to as the parent device. The other device, the lower layer device, represents a device at the lower layer of the hierarchy, which cannot connect directly to the cloud. You can add more lower layer devices to represent your production environment, as needed. Devices at lower layers will also be referred to as child devices.
A downstream device emits data directly to the Internet or to gateway devices (IoT Edge-enabled or not). A child device can be a downstream device or a gateway device in a nested topology.
To create a hierarchy of IoT Edge devices, you will need:
A computer (Windows or Linux) with internet connectivity.
A free or standard tier IoT Hub in Azure.
A Bash shell in Azure Cloud Shell using Azure CLI v2.3.1 with the Azure IoT extension v0.10.6 or higher installed. This tutorial uses the Azure Cloud Shell. If you're unfamiliar with the Azure Cloud Shell, check out a quickstart for details.
- To see your current versions of the Azure CLI modules and extensions, run az version.
A Linux device to configure as an IoT Edge device for each device in your hierarchy. This tutorial uses two devices. If you don't have devices available, you can create Azure virtual machines for each device in your hierarchy using the command below.
Replace the placeholder text in the following command and run it twice, once for each virtual machine. Each virtual machine needs a unique DNS prefix, which will also serve as its name. The DNS prefix must conform to the following regular expression:
az deployment group create \ --resource-group <REPLACE_WITH_YOUR_RESOURCE_GROUP> \ --template-uri "https://raw.githubusercontent.com/Azure/iotedge-vm-deploy/1.4/edgeDeploy.json" \ --parameters dnsLabelPrefix='<REPLACE_WITH_UNIQUE_DNS_FOR_VIRTUAL_MACHINE>' \ --parameters adminUsername='azureuser' \ --parameters authenticationType='sshPublicKey' \ --parameters adminPasswordOrKey="$(< ~/.ssh/id_rsa.pub)" \ --query "properties.outputs.[publicFQDN.value, publicSSH.value]" -o tsv
The virtual machine uses SSH keys for authenticating users. If you are unfamiliar with creating and using SSH keys, you can follow the instructions for SSH public-private key pairs for Linux VMs in Azure.
IoT Edge version 1.4 is preinstalled with this ARM template, saving the need to manually install the assets on the virtual machines. If you are installing IoT Edge on your own devices, see Install Azure IoT Edge for Linux or Update IoT Edge.
A successful creation of a virtual machine using this ARM template will output your virtual machine's
SSHhandle and fully qualified domain name (
FQDN). You will use the SSH handle and either the FQDN or IP address of each virtual machine for configuration in later steps, so keep track of this information. A sample output is pictured below.
You can also find the IP address and FQDN on the Azure portal. For the IP address, navigate to your list of virtual machines and note the Public IP address field. For the FQDN, go to each virtual machine's overview page and look for the DNS name field.
Make sure that the following ports are open inbound for all devices except the lowest layer device: 443, 5671, 8883:
- 443: Used between parent and child edge hubs for REST API calls and to pull docker container images.
- 5671, 8883: Used for AMQP and MQTT.
For more information, see how to open ports to a virtual machine with the Azure portal.
Configure your IoT Edge device hierarchy
IoT Edge devices make up the layers of your hierarchy. This tutorial will create a hierarchy of two IoT Edge devices: the top layer device and its downstream, the lower layer device. You can create additional downstream devices as needed.
To create and configure your hierarchy of IoT Edge devices, you'll use the
iotedge-config tool. This tool simplifies the configuration of the hierarchy by automating and condensing several steps into two:
Setting up the cloud configuration and preparing each device configuration, which includes:
- Creating devices in your IoT Hub
- Setting the parent-child relationships to authorize communication between devices
- Generating a chain of certificates for each device to establish secure communication between them
- Generating configuration files for each device
Installing each device configuration, which includes:
- Installing certificates on each device
- Applying the configuration files for each device
iotedge-config tool will also make the module deployments to your IoT Edge device automatically.
To use the
iotedge-config tool to create and configure your hierarchy, follow the steps below in the Azure CLI:
In the Azure Cloud Shell, make a directory for your tutorial's resources:
Download the release of the configuration tool and configuration templates:
cd ~/nestedIotEdgeTutorial wget -O iotedge_config.tar "https://github.com/Azure-Samples/iotedge_config_cli/releases/download/latest/iotedge_config_cli.tar.gz" tar -xvf iotedge_config.tar
This will create the
iotedge_config_cli_releasefolder in your tutorial directory.
The template file used to create your device hierarchy is the
iotedge_config.yamlfile found in
~/nestedIotEdgeTutorial/iotedge_config_cli_release/templates/tutorial. In the same directory,
deploymentLowerLayer.jsonis a JSON deployment file containing instructions for which modules to deploy to your lower layer device. The
deploymentTopLayer.jsonfile is the same, but for your top layer device, as the modules deployed to each device are not the same. The
device_config.tomlfile is a template for IoT Edge device configurations and will be used to automatically generate the configuration bundles for the devices in your hierarchy.
If you'd like to take a look at the source code and scripts for the
iotedge-configtool, check out the Azure-Samples repository on GitHub.
Open the tutorial configuration template and edit it with your information:
In the iothub section, populate the
iothub_namefields with your information. This information can be found on the overview page of your IoT Hub on the Azure portal.
In the optional certificates section, you can populate the fields with the absolute paths to your certificate and key. If you leave these fields blank, the script will automatically generate self-signed test certificates for your use. If you're unfamiliar with how certificates are used in a gateway scenario, check out the how-to guide's certificate section.
In the configuration section, the
template_config_pathis the path to the
device_config.tomltemplate used to create your device configurations. The
default_edge_agentfield determines what Edge Agent image lower layer devices will pull and from where.
In the edgedevices section, for a production scenario, you can edit the hierarchy tree to reflect your desired structure. For the purposes of this tutorial, accept the default tree. For each device, there is a
device_idfield, where you can name your devices. There is also the
deploymentfield, which specifies the path to the deployment JSON for that device.
You can also manually register IoT Edge devices in your IoT Hub through the Azure portal, Azure Cloud Shell, or Visual Studio Code. To learn how, see the beginning of the end-to-end guide on manually provisioning a Linux IoT Edge device.
You can define the parent-child relationships manually as well. See the create a gateway hierarchy section of the how-to guide to learn more.
Save and close the file:
CTRL + S,
CTRL + Q
Create an outputs directory for the configuration bundles in your tutorial resources directory:
Navigate to your
iotedge_config_cli_releasedirectory and run the tool to create your hierarchy of IoT Edge devices:
cd ~/nestedIotEdgeTutorial/iotedge_config_cli_release ./iotedge_config --config ./templates/tutorial/iotedge_config.yaml --output ./outputs
--outputflag, the tool creates the device certificates, certificate bundles, and a log file in the outputs directory setup earlier.
The configuration tool creates your IoT Edge devices and sets up the parent-child relationships between them. Optionally, it creates certificates for your devices to use. If paths to deployment JSONs are provided, the tool will automatically create these deployments to your devices, but this is not required. Finally, the tool will generate the configuration bundles for your devices and place them in the output directory. For a thorough look at the steps taken by the configuration tool, see the log file in the output directory.
user@Azure:~/nestedIotEdgeTutorial$ sudo ./iotedge_config --config ./templates/tutorial/iotedge_config.yaml --output ./outputs Reading "templates/tutorial/iotedge_config.yaml" Writing logs to "outputs/log_2022-08-31_20-39-42.txt" top-layer └──lower-layer No Root CA specified. Generating self-signed root at "./outputs/certificates/iotedge_config_cli_root.pem". Creating certificates for 2 devices Creating 2 devices in hub contoso-iothub Creating configuration files based on "./templates/tutorial/device_config.toml" for 2 devices. Done! Output located at "/home/user/nestedIotEdgeTutorial/outputs". See README.md in output for install instructions.
Double-check that the topology output from the script looks correct. Once you are satisfied your hierarchy is correctly structured, you are ready to proceed.
Configure the IoT Edge runtime
In addition to the provisioning of your devices, the configuration steps establish trusted communication between the devices in your hierarchy using the certificates you created earlier. The steps also begin to establish the network structure of your hierarchy. The top layer device will maintain internet connectivity, allowing it to pull images for its runtime from the cloud, while lower layer devices will route through the top layer device to access these images.
To configure the IoT Edge runtime, you need to apply the configuration bundles created by the setup script to your devices. The configurations slightly differ between the top layer device and a lower layer device, so be mindful of which device's configuration file you are applying to each device.
Each device needs its corresponding configuration bundle. You can use a USB drive or secure file copy to move the configuration bundles to each device.
Be sure to send the correct configuration bundle to each device.
scp <PATH_TO_CONFIGURATION_BUNDLE> <USER>@<VM_IP_OR_FQDN>:~
:~means that the configuration folder will be placed in the home directory on the virtual machine.
Log on to your virtual machine to apply the configuration bundle to the device:
On each device, unzip the configuration bundle. You'll need to install zip first:
sudo apt install zip unzip ~/<PATH_TO_CONFIGURATION_BUNDLE>/<CONFIGURATION_BUNDLE>.zip
On each device, apply the configuration bundle to the device:
On the top layer device, you will receive a prompt to enter the hostname. On the lower layer device, it will ask for the hostname and parent's hostname. Supply the appropriate IP or FQDN for each prompt. You can use either, but be consistent in your choice across devices. The output of the install script is pictured below.
If you want a closer look at what modifications are being made to your device's configuration file, see the configure IoT Edge on devices section of the how-to guide.
If you completed the above steps correctly, you can check your devices are configured correctly.
Run the configuration and connectivity checks on your devices. For the top layer device:
sudo iotedge check
On a newly provisioned device, you may see an error related to IoT Edge Hub:
× production readiness: Edge Hub's storage directory is persisted on the host filesystem - Error
Could not check current state of edgeHub container
This error is expected on a newly provisioned device because the IoT Edge Hub module isn't running. To resolve the error, in IoT Hub, set the modules for the device and create a deployment. Creating a deployment for the device starts the modules on the device including the IoT Edge Hub module.
For the lower layer device, the diagnostics image needs to be manually passed in the command:
sudo iotedge check --diagnostics-image-name <parent_device_fqdn_or_ip>:443/azureiotedge-diagnostics:1.2
On your top layer device, expect to see an output with several passing evaluations. You may see some warnings about logs policies and, depending on your network, DNS policies.
Once you are satisfied your configurations are correct on each device, you are ready to proceed.
Deploy modules to your devices
The module deployments to your devices were automatically generated when the devices were created. The
iotedge-config-cli tool fed deployment JSONs for the top and lower layer devices after they were created. The module deployment were pending while you configured the IoT Edge runtime on each device. Once you configured the runtime, the deployments to the top layer device began. After those deployments completed, the lower layer device could use the IoT Edge API Proxy module to pull its necessary images.
In the Azure Cloud Shell, you can take a look at the top layer device's deployment JSON to understand what modules were deployed to your device:
In addition the runtime modules IoT Edge Agent and IoT Edge Hub, the top layer device receives the Docker registry module and IoT Edge API Proxy module.
The Docker registry module points to an existing Azure Container Registry. In this case,
REGISTRY_PROXY_REMOTEURL points to the Microsoft Container Registry. By default, Docker registry listens on port 5000.
The IoT Edge API Proxy module routes HTTP requests to other modules, allowing lower layer devices to pull container images or push blobs to storage. In this tutorial, it communicates on port 443 and is configured to send Docker container image pull requests route to your Docker registry module on port 5000. Also, any blob storage upload requests route to module AzureBlobStorageonIoTEdge on port 11002. For more information about the IoT Edge API Proxy module and how to configure it, see the module's how-to guide.
If you'd like a look at how to create a deployment like this through the Azure portal or Azure Cloud Shell, see top layer device section of the how-to guide.
In the Azure Cloud Shell, you can take a look at the lower layer device's deployment JSON to understand what modules were deployed to your device:
You can see under
systemModules that the lower layer device's runtime modules are set to pull from
$upstream:443, instead of
mcr.microsoft.com, as the top layer device did. The lower layer device sends Docker image requests the IoT Edge API Proxy module on port 443, as it cannot directly pull the images from the cloud. The other module deployed to the lower layer device, the Simulated Temperature Sensor module, also makes its image request to
If you'd like a look at how to create a deployment like this through the Azure portal or Azure Cloud Shell, see lower layer device section of the how-to guide.
You can view the status of your modules using the command:
az iot hub module-twin show --device-id <edge_device_id> --module-id '$edgeAgent' --hub-name <iot_hub_name> --query "properties.reported.[systemModules, modules]"
This command will output all the edgeAgent reported properties. Here are some helpful ones for monitoring the status of the device: runtime status, runtime start time, runtime last exit time, runtime restart count.
You can also see the status of your modules on the Azure portal. Navigate to the Devices section of your IoT Hub to see your devices and modules.
Once you are satisfied with your module deployments, you are ready to proceed.
View generated data
The Simulated Temperature Sensor module that you pushed generates sample environment data. It sends messages that include ambient temperature and humidity, machine temperature and pressure, and a timestamp.
You can also view these messages through the Azure Cloud Shell:
az iot hub monitor-events -n <iothub_name> -d <lower-layer-device-name>
iotedge check command to verify the configuration and to troubleshoot errors.
You can run
iotedge check in a nested hierarchy, even if the downstream machines don't have direct internet access.
When you run
iotedge check from the lower layer, the program tries to pull the image from the parent through port 443.
sudo iotedge check --diagnostics-image-name $upstream:443/azureiotedge-diagnostics:1.2
azureiotedge-diagnostics value is pulled from the container registry that's linked with the registry module. This tutorial has it set by default to https://mcr.microsoft.com:
If you're using a private container registry, make sure that all the images (IoTEdgeAPIProxy, edgeAgent, edgeHub, Simulated Temperature Sensor, and diagnostics) are present in the container registry.
Clean up resources
You can delete the local configurations and the Azure resources that you created in this article to avoid charges.
To delete the resources:
Sign in to the Azure portal and select Resource groups.
Select the name of the resource group that contains your IoT Edge test resources.
Review the list of resources contained in your resource group. If you want to delete all of them, you can select Delete resource group. If you want to delete only some of them, you can click into each resource to delete them individually.
In this tutorial, you configured two IoT Edge devices as gateways and set one as the parent device of the other. Then, you demonstrated pulling a container image onto the downstream device through a gateway using the IoT Edge API Proxy module. See the how-to guide on the proxy module's use if you want to learn more.
To learn more about using gateways to create hierarchical layers of IoT Edge devices, see the how-to guide on connecting downstream IoT Edge devices.
To see how Azure IoT Edge can create more solutions for your business, continue on to the other tutorials.