Pagination in Azure Data Lake Storage

Question

When I make API call for listing the paths in ADLS Gen2 using maxResults and Continuation as uri parameters.

Initially I'm getting the correct continuation token returned as response headers but when i try to make subsequent calls for listing the remaining files, sometimes ADLS returns with a continuation token that ends with "=="(lets call it type one) and sometimes if returns with the normal token(type two).Now,when I try to make API call using the first type, ADLS gives me an error and it works fine in the case of the other token.

I searched this before and what I found in one of the answers was that we have to ignore the response in which the token returned ends with "==" and we have to make calls again and again ignoring the continuation token(not using this token in the uri and using the same previous request uri) returned until we get the second(working) type of token.

This is a complex task when it comes to larger amount of files,is there a better solution for this? Is there a better solution for getting the right type of continuation token for making subsequent calls?

The Sample request for making the call is

GET https://storageAccountName.dfs.core.windows.net/sampleDirectory?recursive={recursive}&resource=filesystem&maxresults={maxresults}&continuation={continuation}
The two types of sample continuation tokens returned are

LCJhbGciOiJSUzI1NiIsIng1dCI6ImpTMVhvMU9XRGpfNTJ2Ynd==(Not working one)
LCJhbGciOiJSUzI1NiIsIng1dCI6ImpTMVhvMU9XRGpfNTJ2Ynd (working one)
The Official Documentation for ADLS Gen2 for listing paths is https://learn.microsoft.com/en-us/rest/api/storageservices/datalakestoragegen2/path/list

The link to the stackoverflow answer which I was referring to is
https://stackoverflow.com/questions/56117424/cant-get-adls-gen2-rest-continuation-token-to-work

Answer 1

Hello @MartinJaffer-MSFT , thank you for giving attention to my concern.

Let me explain my problem more clearly with an example-
Lets say we have 4 files in a particular directory and I've set maxResults to 1.
So at the initial call it will give me the first file and a corresponding continuation token as part of the response headers.This answers your 4th question that it is successful always on the first try.
Now lets suppose we are on the 2nd file currently and after the call in response the token is returned.
Now what has been observed is that there are 3 types of token returned:-

Type 1) VBbTpf35gZiDzncYJRgMZGlyZWN0b3J5MS0xFrrXyoP5ur3YAxgAFrDyovnH17bYAwAA
this type of token works fine and gives me the third file if used.

Type 2) VBaupLnSjvHc1dcBGCUYDGRpcmVjdG9yeTEtMRbKxrnfmbu92AMYABaw8qL5x9e22AMAAA==
SCENARIO 1 ) This token should ideally give me the 3rd file but it behaves abnormaly and gives me the 1st file again when used with the "==" in the uri.
SCENARIO 2) When I try to make a call after escaping the "==", it works fine.

Type 3) VBa5ma+h+pryy8EBGCUYDGRpcmVjdG9yeTEtMRaW75jYmLu92AMYABaw8qL5x9e22AMAAA==(you can see the additional '+' in the uri)
SCENARIO 1) This behaves same as it did in Type 2
SCENARIO 2) When i try to escape the "==" from the uri it give me this message "Server encountered an internal error. Please try again after some time.\n"
I think its because of the extra encoded characters in the token.

Now answering to your questions..

1. The api version used is-'2019-02-02'
2. The occurance of this error is 1 in 10.
   Hope the 3rd and 4th questions are being answered by the example.
   A help would be much appreciable.
   Thank You

Answer 2

This issue still occurs

@MartinJaffer-MSFT Would you mind taking a look at this?

Answer 3

@MartinJaffer-MSFT

This does not work at all via the REST API.

Please show your working test with CURL examples to prove this works as expected.

Also the documentation is incorrect and maxResults should be maxresults, as the service has case sensitive query params.

Thanks