Any log or history of IoT Edge Module Deployments and configurations? Security Concerns.

Question

I have looked all over, and I can not find any history of device deployments and configurations.

I can deploy to a IoTEdge device with whatever configuration I wanted, and while there would be logs of the device and modules going up or down, there is nothing that logs by who or what was even deployed. What makes matters worse, is that after you deploy, you have no idea what the previous deployment configuration was. There is no history of deployments being stored. Right now, at best, IF you knew something happened, the only real information that you can find is to connect to the device itself and search trough the docker images for something out of place. The logs on the device wont have the configuration information, but it would show a few lines of it pulling an image and starting/stopping a container.

Real example is, we had a test device sending up data. At some point, we deployed something, and suddenly the code was taking 10x longer to run. This went unnoticed for a while because we were using small datasets. When we went to test it against a large dataset, it was obvious something went wrong. Without having physical access to the device, the best I could do was look trough the data and find when the delay started to happen. From there, I had to pull several images hoping to see variations in the code. The code for this module never really changed though. There might have been a line here or there different, but nothing that would have caused the difference. Once again, we didn't know what version was on there, so we were just guessing on the version that had run on it. I eventually figured out that there was a env variable flag that was (probably) set in the deployment that made a subtle change in the code that caused the huge runtime disparity. I say probably, because we have the normal speed again, but I never saw definitive proof like the original deployment configuration.

Lots of lessons learned from this, but it seems very odd that there are no real logs for the deployments and configurations.

Is there something in the security daemon config.yaml that would keep record of all the deployment changes?

Answer 1

In general most Azure services (Web Apps, VMs, Functions, etc) don't keep a detailed history of what was running inside. Instead we rely on a CI/CD process to provide a robust pipeline and archive of what was deployed, when, and by who. For IoT Edge deployments I'd recommend taking a look at using something like Azure DevOps Tasks to deploy IoT Edge or using the Azure CLI IoT Commandlets to manage deployments from GitHub Actions or other sources.

With a CI/CD pipeline you can choose to archive manifests, container images, enforce approvals before release, create deployment rings, and more!

If you'd like an example, here's a project I have set up with an end-to-end guide:

Answer 2

Hi, can you try enabling the "configurations" diagnostic logs for IoT Hub, and see if that helps?

To enable: https://learn.microsoft.com/en-us/azure/iot-hub/monitor-iot-hub#collection-and-routing

About this log category: https://learn.microsoft.com/en-us/azure/iot-hub/monitor-iot-hub-reference#configurations

I suspect it's not at the level of detail that you need but would love to hear feedback on these diagnostic logs directly. If/when we add improvements it would be here.