Edit

Pagination with the Azure SDK for .NET

  • Article

In this article, you'll learn how to use the Azure SDK for .NET pagination functionality to work efficiently and productively with large data sets. Pagination is the act of dividing large data sets into pages, making it easier for the consumer to iterate through smaller amounts of data. Starting with C# 8, you can create and consume streams asynchronously using Asynchronous (async) streams. Async streams are based on the IAsyncEnumerable<T> interface. The Azure SDK for .NET exposes an implementation of IAsyncEnumerable<T> with its AsyncPageable<T> class.

All of the samples in this article rely on the following NuGet packages:

For the latest directory of Azure SDK for .NET packages, see Azure SDK latest releases.

Pageable return types

Clients instantiated from the Azure SDK for .NET can return the following pageable types.

Type Description
Pageable<T> A collection of values retrieved in pages
AsyncPageable<T> A collection of values asynchronously retrieved in pages

Most of the samples in this article are asynchronous, using variations of the AsyncPageable<T> type. Using asynchronous programming for I/O-bound operations is ideal. A perfect use case is using the async APIs from the Azure SDK for .NET as these operations represent HTTP/S network calls.

Iterate over AsyncPageable with await foreach

To iterate over an AsyncPageable<T> using the await foreach syntax, consider the following example:

async Task IterateSecretsWithAwaitForeachAsync()
{
    AsyncPageable<SecretProperties> allSecrets = client.GetPropertiesOfSecretsAsync();

    await foreach (SecretProperties secret in allSecrets)
    {
        Console.WriteLine($"IterateSecretsWithAwaitForeachAsync: {secret.Name}");
    }
}

In the preceding C# code:

  • The SecretClient.GetPropertiesOfSecretsAsync method is invoked and returns an AsyncPageable<SecretProperties> object.
  • In an await foreach loop, each SecretProperties is asynchronously yielded.
  • As each secret is materialized, its Name is written to the console.

Iterate over AsyncPageable with while

To iterate over an AsyncPageable<T> when the await foreach syntax isn't available, use a while loop.

async Task IterateSecretsWithWhileLoopAsync()
{
    AsyncPageable<SecretProperties> allSecrets = client.GetPropertiesOfSecretsAsync();

    IAsyncEnumerator<SecretProperties> enumerator = allSecrets.GetAsyncEnumerator();
    try
    {
        while (await enumerator.MoveNextAsync())
        {
            SecretProperties secret = enumerator.Current;
            Console.WriteLine($"IterateSecretsWithWhileLoopAsync: {secret.Name}");
        }
    }
    finally
    {
        await enumerator.DisposeAsync();
    }
}

In the preceding C# code:

Iterate over AsyncPageable pages

If you want control over receiving pages of values from the service, use the AsyncPageable<T>.AsPages method:

async Task IterateSecretsAsPagesAsync()
{
    AsyncPageable<SecretProperties> allSecrets = client.GetPropertiesOfSecretsAsync();

    await foreach (Page<SecretProperties> page in allSecrets.AsPages())
    {
        foreach (SecretProperties secret in page.Values)
        {
            Console.WriteLine($"IterateSecretsAsPagesAsync: {secret.Name}");
        }

        // The continuation token that can be used in AsPages call to resume enumeration
        Console.WriteLine(page.ContinuationToken);
    }
}

In the preceding C# code:

Use System.Linq.Async with AsyncPageable

The System.Linq.Async package provides a set of LINQ methods that operate on IAsyncEnumerable<T> type. Because AsyncPageable<T> implements IAsyncEnumerable<T>, you can use System.Linq.Async to query and transform the data.

Convert to a List<T>

Use ToListAsync to convert an AsyncPageable<T> to a List<T>. This method might make several service calls if the data isn't returned in a single page.

async Task ToListAsync()
{
    AsyncPageable<SecretProperties> allSecrets =
        client.GetPropertiesOfSecretsAsync();

    List<SecretProperties> secretList = await allSecrets.ToListAsync();

    secretList.ForEach(secret =>
        Console.WriteLine($"ToListAsync: {secret.Name}"));
}

In the preceding C# code:

  • The SecretClient.GetPropertiesOfSecretsAsync method is invoked and returns an AsyncPageable<SecretProperties> object.
  • The ToListAsync method is awaited, which materializes a new List<SecretProperties> instance.

Take the first N elements

Take can be used to get only the first N elements of the AsyncPageable. Using Take will make the fewest service calls required to get N items.

async Task TakeAsync(int count = 30)
{
    AsyncPageable<SecretProperties> allSecrets =
        client.GetPropertiesOfSecretsAsync();

    await foreach (SecretProperties secret in allSecrets.Take(count))
    {
        Console.WriteLine($"TakeAsync: {secret.Name}");
    }
}

More methods

System.Linq.Async provides other methods that provide functionality equivalent to their synchronous Enumerable counterparts. Examples of such methods include Select, Where, OrderBy, and GroupBy.

Beware client-side evaluation

When using the System.Linq.Async package, beware that LINQ operations are executed on the client. The following query would fetch all the items just to count them:

// ⚠️ DON'T DO THIS! 😲
int expensiveSecretCount =
    await client.GetPropertiesOfSecretsAsync()
        .CountAsync();

Warning

The same warning applies to operators like Where. Always prefer server-side filtering, aggregation, or projections of data if available.

As an observable sequence

The System.Linq.Async package is primarily used to provide observer pattern capabilities over IAsyncEnumerable<T> sequences. Asynchronous streams are pull-based. As their items are iterated over, the next available item is pulled. This approach is in juxtaposition with the observer pattern, which is push-based. As items become available, they're pushed to subscribers who act as observers. The System.Linq.Async package provides the ToObservable extension method that lets you convert an IAsyncEnumerable<T> to an IObservable<T>.

Imagine an IObserver<SecretProperties> implementation:

sealed file class SecretPropertyObserver : IObserver<SecretProperties>
{
    public void OnCompleted() =>
        Console.WriteLine("Done observing secrets");

    public void OnError(Exception error) =>
        Console.WriteLine($"Error observing secrets: {error}");

    public void OnNext(SecretProperties secret) =>
        Console.WriteLine($"Observable: {secret.Name}");
}

You could consume the ToObservable extension method as follows:

IDisposable UseTheToObservableMethod()
{
    AsyncPageable<SecretProperties> allSecrets =
        client.GetPropertiesOfSecretsAsync();

    IObservable<SecretProperties> observable = allSecrets.ToObservable();

    return observable.Subscribe(
        new SecretPropertyObserver());
}

In the preceding C# code:

  • The SecretClient.GetPropertiesOfSecretsAsync method is invoked and returns an AsyncPageable<SecretProperties> object.
  • The ToObservable() method is called on the AsyncPageable<SecretProperties> instance, returning an IObservable<SecretProperties>.
  • The observable is subscribed to, passing in the observer implementation, returning the subscription to the caller.
  • The subscription is an IDisposable. When it's disposed, the subscription ends.

Iterate over pageable

Pageable<T> is a synchronous version of AsyncPageable<T> that can be used with a normal foreach loop.

void IterateWithPageable()
{
    Pageable<SecretProperties> allSecrets = client.GetPropertiesOfSecrets();

    foreach (SecretProperties secret in allSecrets)
    {
        Console.WriteLine($"IterateWithPageable: {secret.Name}");
    }
}

Important

While this synchronous API is available, use the asynchronous API alternatives for a better experience.

See also