question

PapajohnApril-4314 avatar image
0 Votes"
PapajohnApril-4314 asked SumanthMarigowda-MSFT rolled back

Blob Change-Feed Continuation Token is not working as expected

I am using the Java SDK to get blob-modification updates, and I am seeing surprising behavior from the continuationToken. I am storing the continuationToken from the last page of results and using that stored token for my next request, i.e.:

  1. String theCheckpoint = page.getContinuationToken();

  2. Wait a while, modify some blobs in the meantime

  3. results = blobChangefeedClient.getEvents(theCheckpoint);

results will be empty. I would expect results to contain the records of the blob-modifications I did.

The problem seems to be the EndTime field that is in the token, because if I take that token and manually modify the EndTime to be one day out, then the token works as desired and I get back the events that happened.

The question is: why doesn't the blob-change-feed request detect the latest changes when I pass it the continuation token?


After further exploration, I think I can narrow down the problem.
At 15:15 on April 9th, I requested the events from the change feed, passing a blank for a continuation-token. I got this back:
{"CursorVersion":1,"UrlHost":"zuse1hltd1lexists.blob.core.windows.net","EndTime":"2021-04-09T15:00:00Z","CurrentSegmentCursor":{"ShardCursors":[{"CurrentChunkPath":"log/00/2021/04/08/1700/00000.avro","BlockOffset":3054,"EventIndex":0}],"CurrentShardPath":"log/00/2021/04/08/1700/","SegmentPath":"idx/segments/2021/04/08/1700/meta.json"}}

The EndTime seems to render this cursor unusable. In order to make it usable, I have to manually modify the EndTime.

azure-blob-storage
· 2
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Realized that this item did not really have a question, so I just edited and added an explicit question at the end.

0 Votes 0 ·

I posted a related question, because I was retesting today and now I'm seeing different behavior. It seems to work better, but I don't know why I'm seeing different behavior:

https://docs.microsoft.com/en-us/answers/questions/355848/change-feed-continuation-token-why-is-the-endtime.html

0 Votes 0 ·
shivapatpi-MSFT avatar image
0 Votes"
shivapatpi-MSFT answered PapajohnApril-4314 commented

Hello @PapajohnApril-4314 ,
Thanks for your query ! It seems this issue is some what similar to the one reported here in git hub. Kindly have a look at it.

Above issue was reported for NodeJS and I guess it is applicable for Java SDK also as the underlying API will be the same.

Root cause was mentioned as "The SDK returns continuation token with "EndTime": null, when it's returned in the second page, which is because converting a null string to an invalid date object in code"

Let us know if the above helps you out in understanding the issue.

· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

It does seem like a similar issue, in that the checkpoint returned does not work as expected. But there is a difference, and I will update this ticket to make it apparent: in the case of the issue you link to, a null is returned for the EndTime. In my case, the EndTime returned does have an apparently valid value in it, but unfortunately, the results that come back for this endpoint are missing expected values. And if I update the endpoint to be farther out, then the checkpoint starts working more as expected.

Will go ahead and edit the ticket.

0 Votes 0 ·
PapajohnApril-4314 avatar image
0 Votes"
PapajohnApril-4314 answered SumanthMarigowda-MSFT rolled back

I discovered the answer. the EndTime field in the continuation token reflects the end time passed in to the client.getEvents call. So it is necessary for the first call to be either:
client.getEvents()
or
client.getEvents(startTime,null) in order to get a usable cursor token back.

5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.