Event Hub with infinite messages

Jona 475 Reputation points
2023-11-29T18:46:47.84+00:00

Hi everyone,

I'm developing an Python Function App locally which is triggered by messages on Event Hubs. Instead of using Event Hubs' Data Generator, I use its Rest Api to send single message or batches.

I performed 9 Api calls of single events, and I could processed them with the function. However, I realized that every time I save changes in VS Code, the function restarts (like if it were nodemon in Node) and suddenly executes and read messages from Event hub.

¿How can this be posible? The messages were already consumed. And it's been happening all day, the same messages.

could anybody help understand this behaviour?

Best regards

Azure Functions
Azure Functions
An Azure service that provides an event-driven serverless compute platform.
4,661 questions
Azure Event Hubs
Azure Event Hubs
An Azure real-time data ingestion service.
600 questions
0 comments No comments
{count} votes

2 answers

Sort by: Most helpful
  1. JananiRamesh-MSFT 23,651 Reputation points
    2023-11-29T19:14:54.7733333+00:00

    @Jona Thanks for reaching out. It is possible that the messages are being reprocessed every time the function app restarts because the messages are still present in the Event Hub. When a function app is restarted, it will reprocess any messages that are still present in the Event Hub.

    To avoid reprocessing the same messages, you can try the following:

    Use a checkpointing mechanism to keep track of the last processed message. This will ensure that the function app only processes new messages that have not been processed before.

    Configure the Event Hub to retain messages for a shorter period of time. This will ensure that messages are not retained in the Event Hub for a long time and are processed quickly.

    Once the message is sent to the EventHub, it will be retained until the retention period expires any no. of consumers who knows the connection string of the particular EventHub can connect to it and it can read the events.

    Note: it’s not like service bus where the message gets deleted once it is read.

    Reference: https://learn.microsoft.com/en-us/archive/blogs/servicebus/data-retention-in-event-hubs

    do let me know if you have any queries.

    Please 'Accept as answer' and ‘Upvote’ if it helped so that it can help others in the community looking for help on similar topics.

    0 comments No comments

  2. Jona 475 Reputation points
    2023-11-29T19:42:31.85+00:00

    Hi,

    This my hosts.json

    {
      "version": "2.0",
      "logging": {
        "applicationInsights": {
          "samplingSettings": {
            "isEnabled": true,
            "excludedTypes": "Request"
          }
        }
      },
      "extensionBundle": {
        "id": "Microsoft.Azure.Functions.ExtensionBundle",
        "version": "[4.*, 5.0.0)"
      },
      "extensions": {
        "eventHubs": {
          "maxEventBatchSize" : 10,
          "minEventBatchSize" : 5,
          "maxWaitTime" : "00:05:00",            
          "batchCheckpointFrequency" : 1,
          "initialOffsetOptions" : {
              "type" : "fromStart"
          },
          "clientRetryOptions":{
              "mode" : "exponential",
              "tryTimeout" : "00:01:00",
              "delay" : "00:00:00.80"
          }
        }
      }
    }
    

    As you see, the property about the frecuency the checkpoint is made (batchCheckPointFrecuency)

    am I missing something??

    0 comments No comments