My devices don't receive C2D messages anymore from IoT Hub

Question

My devices don't receive C2D messages anymore from IoT Hub

Andrea Previtali 51

Hello everyone,

I've been going crazy for a few days with my IoT Hub and I can't figure out why.

On my devices, I have connected about 60-70 of them, the C2D messaging system doesn't work anymore.

My "system" is very simple, it is based on an embedded Linux distribution and I have a script written in C that is invoked every 10'' to send a teletrics message to IoT Hub, so far so good, everything works correctly and never had any problems.

I use mosquitto and MQTT protocol.

Every x seconds, depending on the device, a Subscribe request is also made via another script written in C.

The connection is successfully made to IoT Hub, but the device no downloads the message, it is as if it did not receive it.

My scripts, both the one for sending telemetry and the one for receiving messages are based on the C codes downloaded from here:

https://learn.microsoft.com/it-it/azure/iot-hub/iot-hub-mqtt-support

My devices authenticate with X.509 certificates and everything has always worked properly, I have been using this system for almost a year.

In the past few days I started having problems on sending messages from cloud to device: from Azure interface the message shows correctly sent and queued, but to the device it is as if it never arrives.

I thought that for some reason the post script might conflict with the subscribe script but I noticed that the problem persists even when running only the subscribe script.

To rule out the possibility that the problem could be associated with the platform, I also used the C program in the Linux Ubuntu environment. The problem persists.

Has something changed in Azure IoT Hub that I missed?

Has anyone experienced similar problems?

I am very concerned because I need to restore this "communication" as soon as possible but I don't know how to do it.

Thanks to everybody.

Andrea Previtali

LeelaRajeshSayana-MSFT 9,751 Reputation points

2023-11-16T00:44:31.2266667+00:00

Hi @Andrea Previtali Greetings! I am following up to see if you got a chance to review the below response. Please let us know if you have any additional questions or need further assistance.

If the response helped, please do click Accept Answer and Yes for the answer provided. Doing so would help other community members with similar issue identify the solution. I highly appreciate your contribution to the community.
LeelaRajeshSayana-MSFT 9,751 Reputation points

2023-11-20T20:00:43.4966667+00:00

Hi @Andrea Previtali We still have not heard back from you. Just wanted to check if the answer provided below was helpful? If it answers your query, please do click Accept Answer and Yes for the answer, as it might be beneficial to other community members reading this thread. And, if you have any further query do let us know. Thank you
Andrea Previtali 51 Reputation points

2023-11-21T07:22:12.2733333+00:00

Hi @LeelaRajeshSayana-MSFT ,

You are right, I didn't answer, but I haven't found a solution to my problem yet.

I was hoping to find a solution and be able to give feedback on it, but I haven't opened the ticket with Microsoft yet.

If you think it is right I can mark the response as "accepted," but for now I have not yet solved my problem.

LeelaRajeshSayana-MSFT 9,751 Reputation points

2023-11-16T00:44:31.2266667+00:00

Hi @Andrea Previtali Greetings! I am following up to see if you got a chance to review the below response. Please let us know if you have any additional questions or need further assistance.

If the response helped, please do click Accept Answer and Yes for the answer provided. Doing so would help other community members with similar issue identify the solution. I highly appreciate your contribution to the community.
LeelaRajeshSayana-MSFT 9,751 Reputation points

2023-11-20T20:00:43.4966667+00:00

Hi @Andrea Previtali We still have not heard back from you. Just wanted to check if the answer provided below was helpful? If it answers your query, please do click Accept Answer and Yes for the answer, as it might be beneficial to other community members reading this thread. And, if you have any further query do let us know. Thank you
Andrea Previtali 51 Reputation points

2023-11-21T07:22:12.2733333+00:00

Hi @LeelaRajeshSayana-MSFT ,

You are right, I didn't answer, but I haven't found a solution to my problem yet.

I was hoping to find a solution and be able to give feedback on it, but I haven't opened the ticket with Microsoft yet.

If you think it is right I can mark the response as "accepted," but for now I have not yet solved my problem.

Answer 1

Sander van de Velde 25,266 MVP

Hello @Andrea Previtali ,

welcome to this moderated Azure community forum.

I expect you are not reusing device registrations, please double check.

There has been one mayor change regarding the IoT Hub.

Microsoft is migrating to the new Digicert TLS certificate. This is announced a few years ago an postponed several times to give users more time to migrate.

Are your device capable to working with this Digicert certificate?

If this is ok, I recommend submitting a support ticket so the Azure helpdesk can check your backend.

This answer is written by myself, not by some AI. If the response helped, do "Accept Answer". If it doesn't work, please let us know the progress. All community members with similar issues will benefit by doing so. Your contribution is highly appreciated.

Andrea Previtali 51 Reputation points

2023-11-10T07:17:56.9+00:00

Hi @Sander van de Velde ,

Very thanks for your quick answer.

Now two months ago I also migrated my IoT Hub to the new Digicert TLS certificate.

I think there were no problems since the devices always stayed connected. I have a few with the old certificate still installed but I am planning an in-place upgrade procedure for several reasons.

The problem is that even the "new" devices, namely the ones that have just been registered on the IoT Hub are giving me problems.

I believe there is no problem at the certificate level since they are the same certificates used by the message sending scripts as well; sending messages to IoT Hub continues to work without problems, I only have problems with C2D reception.

The other strange thing is that still the subscribe program authenticates correctly, or at least it seems to, since I receive an ACK of successful connection. The only strange thing is that it does not "download" queued messages.

Do you have any other suggestion?

I'm not very skilled with this subject.
Sander van de Velde 25,266 Reputation points MVP

2023-11-10T09:35:41.5033333+00:00

Hello @Andrea Previtali ,

thanks for confirming the certificate move.

This is an awkward situation and it's hard to give a straight answer or solution.

Firewalls and closed ports can give similar issues but there is still partially communication.

You can always double-check the code to see how cloud messages are handled. If you see message being de-queued in the Azure IoT Hub:

If the count goes toward 0, messages are delivered and accepted at the device.

If you have a Azure Device Provisioning service in place, you could transfer devices to another IoT Hub, in another region to see if the problem is fixed.

Again, consider submitting a support ticket to rule out the IoT Hub.
Andrea Previtali 51 Reputation points

2023-11-15T11:52:45.89+00:00

Hi @Sander van de Velde ,

Thanks for your quick answer... all the times :)

I took two screenshot similar to the one you send in your last message:

As you can see the device is connected to the IoT Hub but for some reason messages don't reach the device (or it doesn't accept them).

Before I open a ticket to Microsoft I was curious to see if I could do some testing to narrow down the problem.

Do you think there is anything else I can investigate ?

I'm not very skilled in this matter, maybe I can try to "read" the messages in some other way?

Maybe in Python or something like that?
Sander van de Velde 25,266 Reputation points MVP

2023-11-15T12:04:51.5866667+00:00
Hello @Andrea Previtali,

The cloud messages you send is just one of the three ways to communicate from cloud to a device client:

cloud messages (async, no response message)

Desired properties, reported properties (sync. let the client pick up changes and it can report back how these are consumed)

Direct method (sync call to (at that moment) connected device. can time out or receive a response)

All three ways of communication need custom logic on the device.

Has the format changed over time? Perhaps the device is trying to parse the incoming message but it is incompatible with the code?

Perhaps the message is 'corrupt' but the device does not abandon the message so the queue is stuck with that message?

I'm afraid you need to have a developer check out why the C2D cloud message is not received. If the device makes a connection and send device messages to the cloud, it seems the problem lies with logic on the device consuming the messages sent.
Andrea Previtali 51 Reputation points

2023-11-15T14:10:37.2866667+00:00

Hello @Sander van de Velde

I add a detail, maybe we can find the solution quickly because there are some "positive" developments.

I have an IoT Hub replica of my "official" one that I have used in the past for testing. For example before I made the switch to G2_Root Certificate on my official IoT Hub I had tested with that.

On the IoT Hub test everything works, the script I use for reading C2D messages is exactly the same, only the reference to the IoT Hub to connect to changes but otherwise it is exactly the same. So I can read C2D messages from TEST IoT Hub without any problem.

I guess then that the circle narrows around my "official" IoT Hub.

What do you think I could check?

Is there any way to verify that the switch to G2_Root was successful? I remember getting a confirmation message, however, you never know.

The two IoT Hubs are the same, except that one is configured with the free Tier and the other, the "official" one, with the Standard subscription.
Sander van de Velde 25,266 Reputation points MVP

2023-11-15T15:06:25.1533333+00:00

Hello Andrea,

The Azure IoT resources are the last Azure resources migration to the new Digicert TLS certtificate.

As seen in that post, we were able to force and test the migration.

A couple of weeks/months ago, Microsoft started to force the migration automatically.

The TLS certificate used to be visible in the Certificates tab:

You could verify the use of the new Digitcert certificate by making it inaccessible for a DeviceClient to access it. One way is to use a constrained device without cert store. Then you have to 'pin' the correct TLS certificate.

If you suspect the TLS migration to give you issues, you need to submit an Azure support ticket. That team is able to check everything.

Please let us know what the outcome is.

Off topic: If you make a connection to the (production) IoT Hub using the Device Provisioning service, you could force a migration for the troubled devices to another IoT Hub.

My devices don't receive C2D messages anymore from IoT Hub

1 answer

Activity