partition readers communication

Question

Hi,

Assume that we have 32 partition and 32 client under same consumer group read partiotion. Producer sends mllions data to 32 partition. So at the end of boundded message producer sends "run" message this message goes to one of the 32 partition. So when consumer get "run" message it need to know every reader should finish reading related partition. So how can consumer know all of the consumer finished related partition read then it will run "run" command. is there any way to now this.As far as ı know there are no global offset between partitions.

Regard

Answer 1

Hello @Sahsuvaroglu, Muhammed (GE Digital) ,

As per the discussion with internal team, for the receive path, partitions are completely independent. There is no cross-partition coordination or global sequence number. If you need to accumulate a particular group of messages before performing an operation, then you will need to implement the logic to do that. Have you considered using partition keys? If all messages in a group can be stamped with the same partition key, then they will all go to the same partition and therefore the same receiver, which simplifies the problem a lot. Since there are no ordering guarantees between messages sent by different producers, you may still have to deal with the situation where the RUN command is in the middle of the group, but again that’s a simpler problem to solve when all the messages are in one partition.

Hope this info helps.

-----------------------------

• Please don't forget to click on or upvote button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how