Why does the server disconnect during secondary ADU processing?

yoji sato 281

We are building a system based on AzureRTOS and implementing update processing using ADU for secondary devices. Due to memory capacity constraints in this system, the ADU download process to the secondary device is divided into parts, but the process may be disconnected from the server side during the process.

Since the ADU agent driver is called each time a packet is received, we have implemented it so that the data is stored in an internal buffer, and when the buffer is full, it performs the transfer process to the secondary device. As a result, when the buffer becomes full, the ADU agent driver takes longer to process, resulting in disconnections like the one below.

When the processing time exceeds approximately 6 minutes, the MQTT connection will be disconnected from IoTHUB.
When the processing time is less than the above (approximately 30 seconds to several minutes), the HTTP connection is disconnected from the ADU server.

Furthermore, it seems that 2) disconnection also occurs after 1).

What is the reason for the above disconnection? Also, is there a way to prevent such disconnections from occurring?

Best regards,

Yoji Sato Grape Systems Inc.

LeelaRajeshSayana-MSFT 13,466 Reputation points

2024-02-01T17:44:40.77+00:00

Hi @yoji sato Thank you for posting the question here. We will review your use case with the product team and share more feedback on this. Please stay tuned to this thread for updates.
yoji sato 281 Reputation points

2024-02-06T00:22:24.58+00:00

Hello, LeelaRajeshSayana, Thank you for your reply.

How is the product team checking this issue? If it seems like it will take more time, it would be helpful if you could let us know how much time it will take.

Best regards,

Yoji Sato Grape Systems Inc.

1 answer

Bo Chen 596 Reputation points Microsoft Employee

2024-02-18T03:15:30.48+00:00

Hi @yoji sato MQTT and ADU agent uses one thread, and MQTT keep-alive timeout is 240 seconds, if driver take long time to process the packet, the MQTT thread will be blocked, then the MQTT connection will be disconnected. Could you improve the driver logic to avoid taking long time to process the packet or add new thread in driver to process/flash the buffer?
Please sign in to rate this answer.
yoji sato 281 Reputation points

2024-02-19T04:41:44.2033333+00:00

Hello, Bo Chen, Thank you for your reply.

The timeout period of 240 seconds seems to be consistent with disconnecting from IoTHUB (although the time observed here seems to be a little longer).

In addition to the above, there is also a problem where the ADU driver is disconnected from the ADU HTTP server side if it takes about 30 seconds to process, preventing the process from proceeding further. Do you know any reason for this?

Best regards, Yoji Sato Grape Systems Inc.

Bo Chen 596 Reputation points Microsoft Employee

2024-02-20T01:30:53.9433333+00:00

Hi @yoji sato Same issue for HTTP, if the ADU driver takes long time to process/flash data, other threads will not have chance to respond TCP ACK, TCP Window probe, etc. for HTTP connection, then HTTP server will disconnect the connection due to timeout.

yoji sato 281 Reputation points

2024-02-22T06:55:49.9333333+00:00

Hello, Bo Chen,
Thank you for your reply.

As I mentioned before, we would like to use the ADU mechanism to update the firmware of secondary devices connected to the microcontroller running Azure RTOS.

At that time, if the size of the firmware on the secondary device is larger than the buffer size of the microcontroller, the firmware cannot be received all at once. Therefore, it is necessary to repeat the process of receiving data from the ADU server up to the buffer size, transferring that data to the secondary device, and receiving the next data from the ADU server.

By using double buffers (or multi-buffers), it is possible to perform reception processing from the ADU server and transfer processing to the secondary device in parallel. However, communication with the secondary device often uses low-speed communication such as UART, and is generally slower than the network communication speed with the ADU server side. Therefore, it may be necessary to wait for the reception process from the ADU server until the transfer to the secondary device is completed.

The ADU driver's processing is called as a callback function every time data is received from the ADU server, and the processing is repeated for the size of the firmware. Therefore, there is no way to control the calling of the callback function depending on the free status of the microcontroller's buffer or the status of data transfer to the secondary device, and the callback function waits for the completion of data transfer to the secondary device. So it will take a long time at the callback function.

If the ADU mechanism cannot support use cases like this, it will probably not be able to support firmware updates for secondary devices.

Is there any good solution?

Best regards,

Yoji Sato Grape Sywstems Inc.

LeelaRajeshSayana-MSFT 13,466 Reputation points

2024-02-28T21:00:07.71+00:00

Hi @Bo Chen I am checking to see if you have any additional feedback to share on the above ask. I highly appreciate your insights on this.

Bo Chen 596 Reputation points Microsoft Employee

2024-02-29T06:53:43.8233333+00:00

Hi @yoji sato, have you attempted to have the ADU driver directly write the packet data into the secondary device instead of using buffer? If so, what issues have arisen with this method? how long does the driver take to flash one packet into secondary device?
Sign in to comment