Add custom metrics
Applies to: IoT Edge 1.4
Gather custom metrics from your IoT Edge modules in addition to the built-in metrics that the system modules provide. The built-in metrics provide great baseline visibility into your deployment health. However, you may require additional information from custom modules to complete the picture. Custom modules can be integrated into your monitoring solution by using the appropriate Prometheus client library to emit metrics. This additional information can enable new views or alerts specialized to your requirements.
Sample modules repository
See the azure-samples repo for examples of custom modules instrumented to emit metrics. Even if a sample in your language of choice isn't yet available, the general approach may help you.
Consult the best practices from Prometheus docs for general guidance. The following additional recommendations can be helpful for IoT Edge scenarios.
Include the module name at the beginning of metric name to make clear which module has emitted the metric.
Include the IoT hub name or IoT Central application name, IoT Edge device ID, and module ID as labels (also called tags/dimensions) in every metric. This information is available as environment variables to every module started by the IoT Edge agent. The approach is demonstrated by the example in samples repo. Without this context, it's impossible to associate a given metric value to a particular device.
Include an instance ID in the labels. An instance ID can be any unique ID like a GUID that is generated during module startup. Instance ID information can help reconcile module restarts when processing a module's metrics in the backend.
Configure the metrics collector to collect custom metrics
Once a custom module is emitting metrics, the next step is to configure the metrics-collector module to collect and transport custom metrics.
The environment variable
MetricsEndpointsCSV must be updated to include the URL of the custom module's metrics endpoint. When updating the environment variable, be sure to include the system module endpoints as shown in the metric collector configuration example.
By default, a custom module's metrics endpoint does not need to be mapped to a host port to allow the metrics collector to access it. Unless explicitly overridden, on Linux, both modules are started on a user-defined Docker bridge network named azure-iot-edge.
User-defined Docker networks include a default DNS resolver that allows inter-module communication using module (container) names. For example, if a custom module named module1 is emitting metrics on http port 9600 at path /metrics, the collector should be configured to collect from endpoint http://module1:9600/metrics.
Run the following command on the IoT Edge device to test if metrics emitted by a custom module on http port 9600 at path /metrics are accessible:
sudo docker exec replace-with-metrics-collector-module-name curl http://replace-with-custom-module-name:9600/metrics
Add custom visualizations
Once you're receiving custom metrics in Log Analytics, you can create custom visualizations and alerts. The monitoring workbooks can be augmented to add query-backed visualizations.
Every metric is associated with the resource ID of the IoT hub or IoT Central application. That's why you can check if your custom metrics ingested correctly from the Logs page of the associated IoT hub or IoT Central application instead of the backing Log Analytics workspace. Use this basic KQL query to verify:
InsightsMetrics | where Name == 'replace-with-custom-metric-name'
Explore additional metrics visualization options with curated workbooks.