Not receiving ACKs when connection to IoT Hub through an edge device

Hello,

I am connecting to the IoT Hub through an edge device. I am sending some telemetry data to the edge. The packets send are processed on the IoT Edge and stored somewhere. The issue that I am currently having is not getting some ACK packets from the server sometimes which causes a connection reset.

Their are 3 different cases:

1) the device is sending telemetary packet and receiving the ACK from the server after each packet
2) The device send few packets without getting any response from the server, and after a while the server send successive ACK packets.
3) Case number 2 trigger a reset. so the device send a TCP RST flag and then attempt to re-connect to the server.

What happens after triggering the RST flag depends on the version of QOS that I am currently using.

When using QOS 1, the device never recovers from this state and keep going in a loop trying to send RST and re-connect, until it eventually stops any further attempts and it gets stuck somewhere in the code.
When using QOS 0, the device recovers normally after sending the RST flag. However, the issue might happen again after some time.

I am attaching screenshots of the wireshark capture for the three different cases that I described.

my question is, what might be the reason that the server is not responding with ACK packets? Also, what makes the board gets stuck when using QOS1?

Thanks,
Karim

risolis 8,701 Reputation points

2022-05-12T05:49:36.907+00:00

Hello @Karim Walid Abdelazim Elhammady

Thank you for your post!

I want to ask few details so, please see some questions below:

Are you using Quality of service for MQTT?

When you are referring to QOS1,2 and 3... Is this for QoS levels in MQTT?

Looking forward to your feedback,

BR,
Karim Walid Abdelazim Elhammady 16 Reputation points

2022-05-12T06:46:06.393+00:00

Hello @risolis ,

thanks for your reply. yes I am using QOS for MQTT. That's what I meant when referring to it.

Thanks,
Karim
risolis 8,701 Reputation points

2022-05-12T07:02:08.257+00:00

Hello,

Did you configure the HTTP_PROXY and URL PROXY settings?

BR,
Karim Walid Abdelazim Elhammady 16 Reputation points

2022-05-12T16:39:02.75+00:00

I didn't configure any proxy settings

2 answers

risolis 8,701 Reputation points

2022-05-12T07:47:53.22+00:00

Hi @Karim Walid Abdelazim Elhammady

Please take a look at this:

Here is the article where you can get mode details about it.

https://learn.microsoft.com/en-us/azure/iot-dps/iot-dps-mqtt-support

BR,

Please "Accept the answer" if the information helped you. This will help us and others in the community as well.
Please sign in to rate this answer.

1 person found this answer helpful.
Karim Walid Abdelazim Elhammady 16 Reputation points

2022-05-12T16:40:15.797+00:00

I am not using DPS. so I think this might be irrelevant to my case. I think I need to find the main reason for the packets dropped.

risolis 8,701 Reputation points

2022-05-12T17:38:13.197+00:00

Thanks for your answer.

You can also specify MQTT over Web Sockets in the client protocol parameter.

By default, the device SDKs connect to an IoT Hub with the CleanSession flag set to 0 and use QoS 1 for message exchange with the IoT hub. While it's possible to configure QoS 0 for faster message exchange, you should note that the delivery isn't guaranteed nor acknowledged. For this reason, QoS 0 is often referred as "fire and forget".

When a device is connected to an IoT hub, the device SDKs provide methods that enable the device to exchange messages with an IoT hub.

See the default timers for this:

risolis 8,701 Reputation points

2022-05-12T17:43:19.113+00:00

Also, I wondering if you configure the MQTT keep-alive and recall that this can be either the same for each device or diferent.

Finally, Another reason for this drops packet is that the SDK version used has a bug.

Let me know if further details are needed!

BR,

Karim Walid Abdelazim Elhammady 16 Reputation points

2022-05-12T18:01:29.51+00:00

Can I know what SDK version that has the bug? And which SDK version fixes this issue? I am using Version: 6.1.10 ( commit: fafaebe513129e0899a832b0899fa9e69187ee06) (repository link: https://github.com/azure-rtos/netxduo)

risolis 8,701 Reputation points

2022-05-12T19:34:47.223+00:00

If I am not mistaken that SDK version is on preview....

Karim Walid Abdelazim Elhammady 16 Reputation points

2022-05-12T21:57:17.387+00:00

Can you please send me the full ink for this screenshot?

Karim Walid Abdelazim Elhammady 16 Reputation points

2022-05-12T21:57:27.533+00:00

How can I confirm?
Sign in to comment
risolis 8,701 Reputation points

2022-05-13T06:59:00.277+00:00

@Karim Walid Abdelazim Elhammady

Thanks for your patience on this.

I would like to comment you that is expected as well as per design that QoS2 shall be dropped. The SDK version should be either 5.0 version or later.

If you want to double check my answer please go thought this explanation below:

NetX Duo MQTT Basic Operation
MQTT (Message Queue Telemetry Transport) is based on publisher-subscriber model. A client can publish information to other clients through a broker. A client, if interested in a topic, can subscribe to the topic through the broker. A broker is responsible for delivering published messages to its clients who subscribe to the topic. In this publisher-subscriber model, multiple clients may publish data with the same topic. A client will receive a message it publishes if the client subscribes to the same topic.

Depending on the use case, a client may choose one of the 3 QoS levels when publishing a message:
QoS 0: The message is delivered at most once. Messages sent with QoS 0 may be lost.
QoS 1: The message is delivered at least once. Messages sent with QoS 1 may be delivered more than once.
QoS 2: The message is delivered exactly once. Messages sent with QoS 2 is guaranteed to be delivered, with no duplication.
Note

This implementation of MQTT client does not support QoS level 2 messages.

I am hoping that those details were enough to realize for what you have seen on your packet captures. Now you know what to expect or what should be the normal behavior.

Cheers!

Please "Accept the answer" if the information helped you. This will help us and others in the community as well.
Please sign in to rate this answer.

1 person found this answer helpful.
risolis 8,701 Reputation points

2022-05-13T23:59:48.49+00:00

@Karim Walid Abdelazim Elhammady

Do you have any other concern that I can address it? If not, please let me know as well.

Best Regards,

Karim Walid Abdelazim Elhammady 16 Reputation points

2022-05-15T06:06:52.377+00:00

I think I understand the expected behavior, but I still don't know why I am having this issue. (not getting responses from the server)

risolis 8,701 Reputation points

2022-05-15T17:15:07.87+00:00

Hi,

I wonder if I can ask you to disable TCP timestamps and try doing this test again pls.

Once it is done, let me know the end results and I hope this can get solved by doing this.

Looking forward to your feedback,

BR,

shamalsarai-5943 121 Reputation points

2022-05-15T17:26:56.747+00:00

Excellent answers and opinions!

risolis 8,701 Reputation points

2022-05-15T18:20:06.857+00:00

Many thanks!!

Karim Walid Abdelazim Elhammady 16 Reputation points

2022-05-16T05:35:01.427+00:00

Hi,

How can I disable the TCP timestamp? I cannot find it anywhere in the netxduo documentation.

Thanks.
Karim

risolis 8,701 Reputation points

2022-05-16T06:27:56.367+00:00

Hi,

I found a nice article for this as well as this section.

TCP Server Disconnection
Closing the connection is accomplished by calling nx_tcp_socket_disconnect. If no suspension is specified, the server socket sends a RST packet to the client socket and places the socket in the CLOSED state. Otherwise, if a suspension is requested, the full TCP disconnect protocol is performed, as follows:
If the client previously initiated a disconnect request (the server socket has already received a FIN packet, responded with an ACK, and is in the CLOSE WAIT state), NetX Duo promotes the TCP socket state to the LAST ACK state and sends a FIN packet. It then waits for an ACK from the client before completing the disconnect and entering the CLOSED state.

risolis 8,701 Reputation points

2022-05-16T06:28:18.997+00:00

If on the other hand, the server is the first to initiate a disconnect request (the client has not disconnected and the socket is still in the ESTABLISHED state), NetX Duo sends a FIN packet to initiate the disconnect and waits to receive a FIN and an ACK from the client before completing the disconnect and placing the socket in a CLOSED state.
If there are still packets on the socket transmit queue, NetX Duo suspends for the specified timeout to allow those packets to be acknowledged. If the timeout expires, NetX Duo flushes the transmit queue of the server socket.
After the disconnect processing is complete and the server socket is in the CLOSED state, the application must call the nx_tcp_server_socket_unaccept service to end the association of this socket with the server port. Note this service must be called by the application even if nx_tcp_socket_disconnect or nx_tcp_server_socket_accept return an error status. After the nx_tcp_server_socket_unaccept returns, the socket can be used as a client or server socket, or even deleted if it is no longer needed. If accepting another client connection on the same server port is desired, the nx_tcp_server_socket_relisten service should be called on this socket.

risolis 8,701 Reputation points

2022-05-16T06:28:30.177+00:00

https://learn.microsoft.com/en-us/azure/rtos/netx-duo/chapter3

shamalsarai-5943 121 Reputation points

2022-05-16T14:23:03.817+00:00

Excellent answers and opinions!

shamalsarai-5943 121 Reputation points

2022-05-16T14:23:27.39+00:00

Excellent answers and opinions!

risolis 8,701 Reputation points

2022-05-16T17:28:42.847+00:00

Hello @Karim Walid Abdelazim Elhammady

Do you have any further doubt about it?

Best Regards,

Please "Accept the answer" if the information helped you. This will help us and others in the community as well.

risolis 8,701 Reputation points

2022-05-19T03:28:50.127+00:00

Hello @Karim Walid Abdelazim Elhammady

I hope you are doing fine.

I would like to ask if further assistance might be required or if the details stated previously were useful to address this question.

BR,
Sign in to comment