Consume a batch of messages in an Azure Function from an Azure Service Bus

Question

Hello!

I need to send an array of messages for processing through a Service Bus Queue using TypeScript.

However, each item in the array ranges from 4KB to 20KB in size, while the maximum message size allowed is 256KB. This presents a challenge because I cannot simply hardcode the maximum number of items in a single message due to the significant variation in size between the items.

1. I have attempted to create an array of items and add elements to it until the total size of the array reaches around 230KB or 200KB. Initially, I used the 'object-sizeof' library and even tried a simple 'JSON.stringify(message)' method to calculate the size. However, I noticed a significant disparity between the computed size and the size displayed on the Azure Portal. In some cases, the message size exceeded 260KB, despite being coded as 200KB.
2. During my exploration of the service-bus library, I discovered an intriguing approach to compute message size using the AMQP protocol. Unfortunately, importing all the required packages to implement this calculation method seemed excessive. As an alternative, I experimented with the 'tryAddMessage' function in an attempt to determine the maximum number of items I could include in a message.
   • Regrettably, this approach encountered the same issue as the initial approach, as I received an error indicating that the message exceeded the 260KB limit (sometimes even reaching 320KB).

             await items.reduce(async (previousWork, item) => {
               await previousWork;
           
               const message: ServiceBusMessage = { body: item };
           
               if (messageBatch.tryAddMessage({ body: messageToSend.concat([message.body]) })) {
                 messageToSend.push(message.body);
               } else {
                 context.bindings.queueOut.push(messageToSend);
                 messageToSend = [message.body];
               }
           
               messageBatch = await sender.createMessageBatch();
             }, Promise.resolve());
           
             if (messageToSend.length) {
               context.bindings.queueOut.push(messageToSend);
             }
           
           
     

3. I have found another topic discussing the use of the batch approach, which suggests that the function is triggered for each batch, with the batch size defined in the host.json configuration. However, when I implemented this approach, I noticed on the Metrics that a message is created for each individual item, causing the function to be triggered for each item instead of processing the entire array as a single batch. https://www.linkedin.com/pulse/consume-batch-messages-azure-function-from-service-bus-dimitar-iliev
   1. I've sent the batch using sender.sendMessages(messageBatch)
      • The input of the second function is an array

                    export default async function dataTransformer(context: Context, items: Item[]): Promise<void> {
                      const logger = new Logger('DataTransformer');
                      logger.info(`Received ${items.length} items`); // output -> unknown because items is actually an item
                    }
        

      • Host.json
        • I added maxMessagesCount: 10 and didn't work. I also added prefetchCount: 20 and didn't work
        • Maybe, is it not possible to do this in typescript?

I would appreciate any help in understanding two aspects: the message size calculation and why the function does not appear to be triggered with a batch of items.

Thank you!

Answer 1

@Anonymous Thanks for reaching out. Sharing some thought on the above issue.

Yes, you can send the batch messages to service bus using the SDK and the sample code can be found here. If you are using the service bus output binding, then you cannot send the batch message. It will treat it as a single message that will be send to the service bus. When you set maxMessagesCount as 10 then it means that max message that can be retrieve is 10 in a single call and it doesn't guarantee as the maximum message size that retrieve from service bus is the max size of the message. So, if your individual message is around 220 KB and max size supported is 256KB then it will only retrieve once message in a signal call and that could be reason you observe that only one message item is retrieved. Ben has already shared the document for SB performance

In case if you have any specific queries feel free to get back to me.