Help with debugging dropped messages from Routes

Question

So, we are facing a situation where many of our D2C telemetry messages are dropped on a specific Route, after having being received in IoT Hub.

Observing the Logs, we get entries mentioning OperationName: droppedMessage and then in the details of the properties_s object it says InternalError, along with the endpointName (our route name) and the device ID. But other than that, there are no extra information that could help us pinpoint why this behavior is happening.

We've noticed that this starts happening when new devices are added to the IoT Hub, that start transmitting messages, and goes away when devices are removed. Right now, we have around 70 devices on the Hub, but no apparent violation of quotas. At least that's what the metrics show.

Is there anyone that has an idea of what might cause this? Or how we could investigate further why it is caused?

Answer 1

Hello @Nikos Agianniotis ,

welcome to this moderated Azure community forum.

It seems this is noticed in the logging of the IoT Hub?

The only reason I can think of is consumer groups...

Are you routing messages to other services using the default routing mechanism (the event hub compatible endpoint)? Did you reuse the $default for multiple consumers? Please create a separate consumer group for every service consuming the same messages:

Any way, In your case you I recommend creating a support ticket so Microsoft can check the logs of your IoT hub service in the back ground.

Have you already added the metrics collector module to your edge device? This gives some insights in the routing within the edge.

---

If the response helped, do "Accept Answer". If it doesn't work, please let us know the progress. All community members with similar issues will benefit by doing so. Your contribution is highly appreciated.