Azure Eventhub, pausing capture looses events when restarted

Nicholas Ross 0

Setup:

Mysql DB with Java front end sending compressed and encrypted messaged to eventhub. All fine here. EventHub Capture is every 5 mins sending Avro with all messages to Storage account.

For maintenance and upgrades it is sometimes necessary to pause Capture, the issue we appear to have is when we switch on Capture we loose the first set of messages/ those that are stored on the Eventhub between capture off and on. The Avros in size do not exceed the file size to be transmitted and the messages are well within purge date/time of keeping messages.

This obviously gives us an issue where we miss critical messages. Not sure what additional info is required to investigate but some pointers would be very helpful please.

Thanks

Nick

PRADEEPCHEEKATLA-MSFT 89,646 Reputation points Microsoft Employee

2023-09-18T10:16:55.62+00:00

@Nicholas Ross - Just checking in to see if the below answer helped. If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.
PRADEEPCHEEKATLA-MSFT 89,646 Reputation points Microsoft Employee

2023-09-21T09:56:29.3066667+00:00

@Nicholas Ross - We received your feedback that the answer provided on the thread was not helpful.

Kindly let us know what we could have done better to improve the answer and make your engagement experience good. We are here to help you and strive to make your experience better and greatly value your feedback.

I have provided a detailed answer which has clear steps for which you are looking for. If you wish, you may re-surveying/rating for the engagement you received on the thread. Your feedback is very important to us.

Looking forward to you reply. Much appreciate your feedback!

Regards,

PRADEEPCHEEKATLA-MSFT
PRADEEPCHEEKATLA-MSFT 89,646 Reputation points Microsoft Employee

2023-09-28T08:32:59.5533333+00:00

@Nicholas Ross - Just checking in to see if the below answer helped. If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.

1 answer

PRADEEPCHEEKATLA-MSFT 89,646 Reputation points Microsoft Employee

2023-09-11T12:09:42.7733333+00:00

@Nicholas Ross - Thanks for the question and using MS Q&A platform.

It sounds like you are experiencing data loss when you pause and resume Event Hub Capture. This can happen if there are messages in the Event Hub that are not captured before Capture is paused, and those messages are lost when Capture is resumed.

One way to avoid this issue is to use Event Hub checkpoints. Checkpoints allow you to keep track of the last message that was successfully processed by your consumer application. When you resume Capture, you can use the checkpoint to start processing messages from where you left off.

Azure Event Hubs Checkpoint Store can be used for storing checkpoints while processing events from Azure Event Hubs.To use checkpoints, you need to enable them in your consumer application. Here is an example of how to use checkpoints in Java: Azure Event Hubs Checkpoint Store client library for Java - version 1.16.9.

Hope this helps. Do let us know if you any further queries.

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.
Please sign in to rate this answer.
Nicholas Ross 0 Reputation points

2023-09-11T14:56:44.1066667+00:00

Thanks for your reply, this may solve the issue if I manage to get it setup will investigate BUT the messages are all from when the capture has been turned off. The first batch (avro of multiple encrypted compressed messages) sending to container just does not seem to happen.

PRADEEPCHEEKATLA-MSFT 89,646 Reputation points Microsoft Employee

2023-09-12T10:43:24.8066667+00:00

@Nicholas Ross - If I understand correctly, you are saying that the first batch of messages that are sent to the Event Hub after Capture is turned back on are not being captured by Event Hub Capture. Is that correct?

If that is the case, one possible reason for this issue is that the messages are being sent to the Event Hub before Capture is fully initialized and ready to capture messages. When you turn on Capture, it may take some time for it to initialize and start capturing messages. During this time, any messages that are sent to the Event Hub may not be captured by Capture.

To avoid this issue, you can try delaying the sending of messages to the Event Hub until after Capture is fully initialized and ready to capture messages. You can use the EventProcessorHost class in your consumer application to monitor the status of Capture and wait until it is ready before sending messages.

Here is an example of how to use EventProcessorHost to monitor the status of Capture:

EventProcessorHost host = new EventProcessorHost( EventProcessorHost.createHostName("my-host"), "my-event-hub-path", "my-consumer-group", "my-event-hub-connection-string", new AzureStorageCheckpointLeaseManager("my-storage-account-connection-string", "my-container-name")); host.registerEventProcessor(MyEventProcessor.class, new EventProcessorOptions().setCheckpoint(true)); while (!host.getPartitionManager().getHasStarted()) { Thread.sleep(1000); }

In this example, the while loop waits until the partition manager has started, which indicates that Capture is ready to capture messages. Once the loop exits, you can send messages to the Event Hub.

I hope this helps! Let me know if you have any further questions or if there is anything else I can assist you with.
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

Share via

Azure Eventhub, pausing capture looses events when restarted

1 answer

Your answer