Transient error and retry policy for Service Bus triggered Azure Function

Question

Hi All,

I have a requirement as below:

1. There is a Service Bus Topic which is the source
2. Azure Function is SB triggered
3. At step 1, message is going to Function
4. Step 2 if for calling a 3rd party API for some data enrichment
5. Finally data is POSTed to some API

As depicted in the diagram if connectivity between function and API is lost, the function will fail and need to be retried exponentially. But during that retry the message is lost in SB and we are not saving the message anywhere. during retry from where will it fetch the message?

I implemented something like in catch block of function we are abandoning the message 5 times and in case of >5 retries, I'm deadlettering the message.

Here this retry is not controlled. I want to retry at exponential interval. How I can achieve that?

Regards,

Partha

Answer 1

Hello Pratim Das, Partha C,

Welcome to the Microsoft Q&A and thank you for posting your questions here.

Problem

I understand that you would like to implement controlled exponential retries if the function fails due to connectivity issues with the API, with the situation at hand you have abandons the message up to 5 times and then deadletters it if retries exceed 5, but this approach lacks controlled exponential retry intervals.

Solution

To solve the issue based on your requirements and ensure controlled exponential retries for your Azure Function triggered by a Service Bus Topic, you can utilize Azure's built-in retry policies and implement additional logic for better handling of retries and message persistence. There are couple of things you will need to fine tune in your configurations as follows:

Firstly, you need to configure the retry policy in the host.json file of your Azure Function. This configuration ensures that your function will retry with an exponential backoff strategy.

{
  "version": "2.0",
  "extensions": {
    "serviceBus": {
      "messageHandlerOptions": {
        "maxAutoRenewDuration": "00:30:00",
        "maxConcurrentCalls": 16,
        "autoComplete": false
      }
    }
  },
  "retry": {
    "strategy": "exponentialBackoff",
    "maxRetryCount": 5,
    "minimumInterval": "00:00:05",
    "maximumInterval": "00:15:00"
  }
}

Secondly, in your Azure Function, handle the message processing and implement logic to manage retries. Use the host.json configuration to handle the retries automatically.

using Microsoft.Azure.WebJobs;
using Microsoft.Extensions.Logging;
using System;
using System.Net.Http;
using System.Threading.Tasks;
public static class ProcessMessageFunction
{
    private static readonly HttpClient httpClient = new HttpClient();
    [FunctionName("ProcessMessageFunction")]
    public static async Task Run([ServiceBusTrigger("mytopic", "mysubscription", Connection = "ServiceBusConnectionString")] string mySbMsg, ILogger log)
    {
        log.LogInformation($"C# ServiceBus topic trigger function processed message: {mySbMsg}");
        try
        {
            // Call 3rd party API for data enrichment
            var enrichedData = await EnrichData(mySbMsg);
            // Post enriched data to another API
            await PostData(enrichedData);
            // Complete the message processing
            // If using manual completion, use context.CompleteAsync()
        }
        catch (Exception ex)
        {
            log.LogError($"Exception: {ex.Message}");
            throw; // This will trigger the retry mechanism
        }
    }
    private static async Task<string> EnrichData(string data)
    {
        // Implement your data enrichment logic
        return await Task.FromResult(data);
    }
    private static async Task PostData(string data)
    {
        var response = await httpClient.PostAsync("https://api.example.com/endpoint", new StringContent(data));
        response.EnsureSuccessStatusCode();
    }
}

NOTE:

If the message fails after the maximum retries, it will be automatically deadlettered based on the configured policy in Service Bus. Ensure your Service Bus topic is configured to handle deadlettering appropriately. Also, during the retry process, Azure Functions and Service Bus handle message persistence. The message is not lost; it remains in the queue and is retried based on the configuration. Ensure autoComplete is set to false in your host.json to manually complete the message only after successful processing.

Finally, if you prefer to manually control message completion, you can modify your function to use MessageReceiver and manually complete or abandon messages.

using Microsoft.Azure.ServiceBus;
using Microsoft.Azure.WebJobs;
using Microsoft.Extensions.Logging;
using System;
using System.Net.Http;
using System.Text;
using System.Threading.Tasks;
public static class ProcessMessageFunction
{
    private static readonly HttpClient httpClient = new HttpClient();
    [FunctionName("ProcessMessageFunction")]
    public static async Task Run([ServiceBusTrigger("mytopic", "mysubscription", Connection = "ServiceBusConnectionString")] Message message, ILogger log, MessageReceiver messageReceiver)
    {
        string mySbMsg = Encoding.UTF8.GetString(message.Body);
        log.LogInformation($"C# ServiceBus topic trigger function processed message: {mySbMsg}");
        try
        {
            // Call 3rd party API for data enrichment
            var enrichedData = await EnrichData(mySbMsg);
            // Post enriched data to another API
            await PostData(enrichedData);
            // Manually complete the message processing
            await messageReceiver.CompleteAsync(message.SystemProperties.LockToken);
        }
        catch (Exception ex)
        {
            log.LogError($"Exception: {ex.Message}");
            // Abandon the message to trigger retry
            await messageReceiver.AbandonAsync(message.SystemProperties.LockToken);
            throw; // This will trigger the retry mechanism
        }
    }
    private static async Task<string> EnrichData(string data)
    {
        // Implement your data enrichment logic
        return await Task.FromResult(data);
    }
    private static async Task PostData(string data)
    {
        var response = await httpClient.PostAsync("https://api.example.com/endpoint", new StringContent(data));
        response.EnsureSuccessStatusCode();
    }
}

References

Kindly use the provided additional resources by the right hand of this pad for more reading and references.

Accept Answer

I hope this is helpful! Do not hesitate to let me know if you have any other questions.

** Please don't forget to close up the thread here by upvoting and accept it as an answer if it is helpful ** so that others in the community facing similar issues can easily find the solution.

Best Regards,

Sina Salam