Packet pool is depleted when too much messages are sent via the azure RTOS C Iot sdk

Grégoire DEFOY 20

When rapid firing telemetry messages with nx_azure_iot_hub_client_telemetry_message_create and

nx_azure_iot_publish_mqtt_packet, I noticed that when the memory allocation fails, the packet pool that the IoT client is using is depleted and no packets are released, is this expected behaviour ? what can I do to prevent this and get back the memory ? Is this due to Qos 1 that will not release packets until they are sent ?

This issue happens whether I send packets with my cellular or wi-fi interface so I don't think this is an issue caused by underlying drivers

LeelaRajeshSayana-MSFT 13,471 Reputation points

2023-04-27T19:15:08.7566667+00:00

Hi @Grégoire DEFOY Greetings! Welcome to Microsoft Q&A forum. Thank you for posting the question here. We are reviewing this with our SMEs and get back to you with more updates on this.
Bo Chen 596 Reputation points Microsoft Employee

2023-04-28T01:33:19.7466667+00:00

Hi @Grégoire DEFOY thanks for asking the question. This issue may be caused by underlying drivers. The driver transmits the data from the NX_PACKET buffer, the driver should use the nx_packet_transmit_release service to release the packet to the available packet pool. Could you check if your driver releases the packet? You may refer to the User Guide and RAM driver code for releasing packet. Thanks.
Grégoire DEFOY 20 Reputation points

2023-04-28T08:14:44.2966667+00:00
Hi, I have checked the wi-fi driver, the nx_packet_transmit_release function is called, for my cellular driver, the nx_ppp stack is called and I suppose that is already releases packets. This issue only happens when using the azure-sdk-for-c functions. I am able to use the underlying drivers for transferring files and they do not deplete the packet pool. Note that the problems with the IoT functions only happens at a really fast transfer rate, when waiting for one second between messages, this does not happen. But when using the IoT in conjunction with a file transfer, then the issue may happen.

I get these logs when the pool is depleted:

azure_iot: 327683 - [ERROR] azure_iot: 327683 - Mqtt client send fail: PUBLISH FAIL status: %d azure_iot: 327683 - 65542 azure_iot: 327683 - iot_client: [Telementry] Could not publish packet. Error was NX::0x10006

I can see that this error is defined in the sdk's stack.

Then the MQTT stacks bricks itself and these logs keep showing in this order :

azure_iot: 327683 - [ERROR] azure_iot: 327683 - Create publish packet failed azure_iot: 327683 - azure_iot: 327683 - [ERROR] azure_iot: 327683 - Create telemetry data fail azure_iot: 327683 - azure_iot: 327683 - [ERROR] azure_iot: 327683 - Security Message telemetry message create failed, error=%d azure_iot: 327683 - 1 azure_iot: 327683 - azure_iot: 327683 - [ERROR] azure_iot: 327683 - Failed to send security message, error=%d azure_iot: 327683 - 1 azure_iot: 327683 -
Grégoire DEFOY 20 Reputation points

2023-05-02T13:58:18.92+00:00

Hello again, I found out that if I check the number of available packets in the pool, and do not allocate a telemetry message if there are less than 2 packets then the mqtt client stack will not stop working. I guess that this is due to the fact that the QoS is set to 1 and the mqtt stack allocate another packet for retransmissions. I will try to separate the packet pools and give the IoT instance its own packet pool. but as my device is already memory constrained, I don't think that this is a good fix as it will remove packets that could have been useful elsewhere.

I also checked again with the underlying driver for Wi-Fi, and all the sent packets are freed with nx_packet_transmit_release. Is it possible that a call to nx_packet_allocate or nx_packet_release could corrupt the entire packet pool ?

However as I said, when waiting for a second between IoT messages, the problem doesn't occur, could this be due to a thread priority problem ?

BR
Grégoire DEFOY 20 Reputation points

2023-05-04T13:59:29.23+00:00

Hi,

Thank you for your answer, I will go with checking the availiable packets before a transmission.

Thank your for your feedback and your time.
LeelaRajeshSayana-MSFT 13,471 Reputation points

2023-05-16T17:04:55.1633333+00:00

@Grégoire DEFOY We still have not heard back from you. Just wanted to check if the answer provided below was helpful? If it answers your query, please do click Accept Answer and Yes for the answer, as it might be beneficial to other community members reading this thread. And, if you have any further query do let us know. Thank you

Accepted answer

Bo Chen 596 Reputation points Microsoft Employee

2023-05-04T02:33:55.2533333+00:00

Hi @Grégoire DEFOY, Thanks for sharing more information. Sending one telemetry message, two packets will be used, nx_azure_iot_hub_client_telemetry_message_create allocates the packet (first packet), then MQTT client (QoS1) copies one packet (second packet), the second packet will not be released until MQTT PUB ACK response is received. The first packet will be added TCP header, IP header and sent out in driver (call nx__packet__transmit_release() to mark the packet has been transmitted*.*), then the first packet will not be released until TCP ACK is received. In your scenario, the issue is to send telemetry message too fast with limited packets, it is not a thread priority problem. As you tested, you may increase the packet count, add the check for available packets before allocating a telemetry message, or slow down telemetry transmission.
Please sign in to rate this answer.

1 person found this answer helpful.

0 comments No comments
Sign in to comment

Packet pool is depleted when too much messages are sent via the azure RTOS C Iot sdk

0 additional answers