Edit

Configurable retry logic core APIs in SqlClient

  • Article

Applies to: .NET Framework .NET .NET Standard

Download ADO.NET

If the built-in retry logic providers don't cover your needs, you can create your own custom providers. You can then assign those providers to a SqlConnection or SqlCommand object to apply your custom logic.

The built-in providers are designed around three interfaces that can be used to implement custom providers. Custom retry providers can then be used in the same way as internal retry providers on a SqlConnection or SqlCommand:

  1. SqlRetryIntervalBaseEnumerator: Generates a sequence of time intervals.
  2. SqlRetryLogicBase: Retrieves the next time interval for a given enumerator, if the number of retries has not been exceeded and a transient condition is met.
  3. SqlRetryLogicBaseProvider: Applies retry logic to connection and command operations.

Caution

By implementing a custom retry logic provider, you're in charge of all aspects, including concurrency, performance, and exception management.

Example

The implementation in this sample is as simple as possible to demonstrate step-by-step customization. It doesn't include advanced practices like thread safety, async, and concurrency. For a deep-dive into a real implementation, you can study the pre-defined retry logic in the Microsoft.Data.SqlClient GitHub repository.

  1. Define custom configurable retry logic classes:

    • Enumerator: Define a fixed sequence of time intervals and extend the acceptable range of times from two minutes to four minutes.
    public class CustomEnumerator : SqlRetryIntervalBaseEnumerator
{
    // Set the maximum acceptable time to 4 minutes
    private readonly TimeSpan _maxValue = TimeSpan.FromMinutes(4);

    public CustomEnumerator(TimeSpan timeInterval, TimeSpan maxTime, TimeSpan minTime)
        : base(timeInterval, maxTime, minTime) {}

    // Return fixed time on each request
    protected override TimeSpan GetNextInterval()
    {
        return GapTimeInterval;
    }

    // Override the validate method with the new time range validation
    protected override void Validate(TimeSpan timeInterval, TimeSpan maxTimeInterval, TimeSpan minTimeInterval)
    {
        if (minTimeInterval < TimeSpan.Zero || minTimeInterval > _maxValue)
        {
            throw new ArgumentOutOfRangeException(nameof(minTimeInterval));
        }

        if (maxTimeInterval < TimeSpan.Zero || maxTimeInterval > _maxValue)
        {
            throw new ArgumentOutOfRangeException(nameof(maxTimeInterval));
        }

        if (timeInterval < TimeSpan.Zero || timeInterval > _maxValue)
        {
            throw new ArgumentOutOfRangeException(nameof(timeInterval));
        }

        if (maxTimeInterval < minTimeInterval)
        {
            throw new ArgumentOutOfRangeException(nameof(minTimeInterval));
        }
    }
}
    • Retry logic: Implement retry logic on any command that isn't part of an active transaction. Lower the number of retries from 60 to 20.
    public class CustomRetryLogic : SqlRetryLogicBase
{
    // Maximum number of attempts
    private const int maxAttempts = 20;

    public CustomRetryLogic(int numberOfTries,
                             SqlRetryIntervalBaseEnumerator enumerator,
                             Predicate<Exception> transientPredicate)
    {
        if (!(numberOfTries > 0 && numberOfTries <= maxAttempts))
        {
            // 'numberOfTries' should be between 1 and 20.
            throw new ArgumentOutOfRangeException(nameof(numberOfTries));
        }

        // Assign parameters to the relevant properties
        NumberOfTries = numberOfTries;
        RetryIntervalEnumerator = enumerator;
        TransientPredicate = transientPredicate;
        Current = 0;
    }

    // Prepare this object for the next round
    public override void Reset()
    {
        Current = 0;
        RetryIntervalEnumerator.Reset();
    }

    public override bool TryNextInterval(out TimeSpan intervalTime)
    {
        intervalTime = TimeSpan.Zero;
        // First try has occurred before starting the retry process. 
        // Check if retry is still allowed
        bool result = Current < NumberOfTries - 1;

        if (result)
        {
            // Increase the number of attempts
            Current++;
            // It's okay if the RetryIntervalEnumerator gets to the last value before we've reached our maximum number of attempts.
            // MoveNext() will simply leave the enumerator on the final interval value and we will repeat that for the final attempts.
            RetryIntervalEnumerator.MoveNext();
            // Receive the current time from enumerator
            intervalTime = RetryIntervalEnumerator.Current;
        }
        return result;
    }
}
    • Provider: Implements a retry provider that retries on synchronous operations without a Retrying event. Adds TimeoutException to the existing SqlException transient exception error numbers.
    public class CustomProvider : SqlRetryLogicBaseProvider
{
    // Preserve the given retryLogic on creation
    public CustomProvider(SqlRetryLogicBase retryLogic)
    {
        RetryLogic = retryLogic;
    }

    public override TResult Execute<TResult>(object sender, Func<TResult> function)
    {
        // Create a list to save transient exceptions to report later if necessary
        IList<Exception> exceptions = new List<Exception>();
        // Prepare it before reusing
        RetryLogic.Reset();
        // Create an infinite loop to attempt the defined maximum number of tries
        do
        {
            try
            {
                // Try to invoke the function
                return function.Invoke();
            }
            // Catch any type of exception for further investigation
            catch (Exception e)
            {
                // Ask the RetryLogic object if this exception is a transient error
                if (RetryLogic.TransientPredicate(e))
                {
                    // Add the exception to the list of exceptions we've retried on
                    exceptions.Add(e);
                    // Ask the RetryLogic for the next delay time before the next attempt to run the function
                    if (RetryLogic.TryNextInterval(out TimeSpan gapTime))
                    {
                        Console.WriteLine($"Wait for {gapTime} before next try");
                        // Wait before next attempt
                        Thread.Sleep(gapTime);
                    }
                    else
                    {
                        // Number of attempts has exceeded the maximum number of tries
                        throw new AggregateException("The number of retries has exceeded the maximum number of attempts.", exceptions);
                    }
                }
                else
                {
                    // If the exception wasn't a transient failure throw the original exception
                    throw;
                }
            }
        } while (true);
    }

    public override Task<TResult> ExecuteAsync<TResult>(object sender, Func<Task<TResult>> function, CancellationToken cancellationToken = default)
    {
        throw new NotImplementedException();
    }

    public override Task ExecuteAsync(object sender, Func<Task> function, CancellationToken cancellationToken = default)
    {
        throw new NotImplementedException();
    }
}

  2. Create a retry provider instance consisting of the defined custom types:

    public static SqlRetryLogicBaseProvider CreateCustomProvider(SqlRetryLogicOption options)
{
    // 1. create an enumerator instance
    CustomEnumerator customEnumerator = new CustomEnumerator(options.DeltaTime, options.MaxTimeInterval, options.MinTimeInterval);
    // 2. Use the enumerator object to create a new RetryLogic instance
    CustomRetryLogic customRetryLogic = new CustomRetryLogic(5, customEnumerator, (e) => TransientErrorsCondition(e, options.TransientErrors));
    // 3. Create a provider using the RetryLogic object
    CustomProvider customProvider = new CustomProvider(customRetryLogic);
    return customProvider;
}
    • The following function will evaluate an exception by using the given list of retryable exceptions and the special TimeoutException exception to determine if it's retryable:
    // Return true if the exception is a transient fault.
private static bool TransientErrorsCondition(Exception e, IEnumerable<int> retriableConditions)
{
    bool result = false;

    // Assess only SqlExceptions
    if (retriableConditions != null && e is SqlException ex)
    {
        foreach (SqlError item in ex.Errors)
        {
            // Check each error number to see if it is a retriable error number
            if (retriableConditions.Contains(item.Number))
            {
                result = true;
                break;
            }
        }
    }
    // Other types of exceptions can also be assessed
    else if (e is TimeoutException)
    {
        result = true;
    }
    return result;
}

  3. Use the customized retry logic:

    • Define the retry logic parameters:
    // Define the retry logic parameters
var options = new SqlRetryLogicOption()
{
    // Tries 5 times before throwing an exception
    NumberOfTries = 5,
    // Preferred gap time to delay before retry
    DeltaTime = TimeSpan.FromSeconds(1),
    // Maximum gap time for each delay time before retry
    MaxTimeInterval = TimeSpan.FromSeconds(20),
    // SqlException retriable error numbers
    TransientErrors = new int[] { 4060, 1024, 1025}
};
    • Create a custom retry provider:
    // Create a custom retry logic provider
SqlRetryLogicBaseProvider provider = CustomRetry.CreateCustomProvider(options);

    // Assumes that connection is a valid SqlConnection object 
// Set the retry logic provider on the connection instance
connection.RetryLogicProvider = provider;
// Establishing the connection will trigger retry if one of the given transient failure occurs.
connection.Open();

Note

Don't forget to enable the configurable retry logic switch before using it. For more information, see Enable configurable retry logic.

See also