Durability of Azure Durable functions, is the 10 minutes still apply for azure consumption plan

Question

I want to create an azure durable function which runs daily, where it read all the data from SharePoint lists and populate the reporting SharePoint lists. some lists in the future might have 2 million records, and inside the azure durable functions i am using PnP core SDK to query the data, where PnP core allow us to query the data using pagination, so we can get the 2 million records. Here what I have built so far inside my visual studio project (for simplicity I am currently getting only the list info (Title, Template & ItemCount), while on reality I need to get all the list items):- ActivityFunction.cs:-

using Microsoft.Azure.WebJobs;
using PnP.Core.Model.SharePoint;
using PnP.Core.Services;
using Microsoft.Extensions.Logging;
using Microsoft.Azure.WebJobs.Extensions.DurableTask;


namespace SPDashBoard
{
    public class ActivityFunctions
    {
        private readonly IPnPContextFactory pnpContextFactory;

        public ActivityFunctions(IPnPContextFactory pnpContextFactory)
        {
            this.pnpContextFactory = pnpContextFactory;
        }

        [FunctionName(nameof(GetListInfo))]
        public async Task<IListDetails> GetListInfo([ActivityTrigger] string input,ILogger log)
        {
            log.LogWarning("About to call GetListInfo activity!");
            using (var objPnPContext = await pnpContextFactory.CreateAsync("Default"))
            {
                IList lstTarget = await objPnPContext.Web.Lists.GetByTitleAsync(input, l => l.Title,
                    l => l.TemplateType, log => log.ItemCount);
                await Task.Delay(1000);
                return new IListDetails()
                {
                    Title = lstTarget.Title,
                    TemplateType = lstTarget.TemplateType.ToString(),
                    ItemCount = lstTarget.ItemCount
                };


            }
        

        }



    }
}

OrchestrationFunctions.cs:-

using Microsoft.Azure.WebJobs;
using Microsoft.Azure.WebJobs.Extensions.DurableTask;
using Microsoft.Extensions.Logging;
using System;
using System.Collections.Generic;
using System.Threading.Tasks;

namespace SPDashBoard
{
    public class OrchestratorFunctions
    {
        [FunctionName("GetListsDetailsChaining")]
        public async Task<List<IListDetails>> GetListsDetailsChaining(
            [Microsoft.Azure.WebJobs.Extensions.DurableTask.OrchestrationTrigger] IDurableOrchestrationContext context,ILogger log)
        {
            log = context.CreateReplaySafeLogger(log);
            string[] listArr = new string[] { "WorkOrders", "Technicians" };
            List<IListDetails> listDetails = new List<IListDetails>();
            foreach (var strList in listArr) 
            {
                IListDetails lstDetails = await context.CallActivityAsync<IListDetails>(nameof(ActivityFunctions.GetListInfo), strList);
                listDetails.Add(lstDetails);
            
            }
            return listDetails;



        }

    }
}

Functions.cs

using Microsoft.Azure.WebJobs;
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using Microsoft.Azure.WebJobs.Extensions;
using Microsoft.Azure.Functions.Worker;
using Microsoft.Azure.WebJobs.Extensions.DurableTask;
using Microsoft.Extensions.Logging;


namespace SPDashBoard
{
    public static class Functions
    {

        [FunctionName("GetListsDetailsChaining_SchduleStart")]
        public static async Task<string> GetListsDetailsChaining_SchduleStart( [TimerTrigger("0 */5 * * * *")] TimerInfo myTimer,  //[TimerTrigger("0 0 0 * * *")] TimerInfo myTimer,
            [Microsoft.Azure.WebJobs.Extensions.DurableTask.DurableClient] IDurableOrchestrationClient starter,ILogger log)
        {
            string instanceId = string.Empty;
            log.LogWarning("Starting the http starter Function = GetListDetails");
            instanceId = await starter.StartNewAsync(nameof(OrchestratorFunctions.GetListsDetailsChaining), null);
            DurableOrchestrationStatus durableOrchestrationStatus = await starter.GetStatusAsync(instanceId);
            while (durableOrchestrationStatus.RuntimeStatus != OrchestrationRuntimeStatus.Completed)
            {
                await Task.Delay(200);
                durableOrchestrationStatus = await starter.GetStatusAsync(instanceId);


            }

            return "done";




        }



    }
}

Startup.cs:-

using Microsoft.Azure.Functions.Extensions.DependencyInjection;
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.DependencyInjection;
using PnP.Core.Auth;
using System.Security.Cryptography.X509Certificates;

[assembly: FunctionsStartup(typeof(SPDashBoard.Startup))]
namespace SPDashBoard
{
    class Startup : FunctionsStartup
    {
        public override void Configure(IFunctionsHostBuilder builder)
        {

            var config = builder.GetContext().Configuration;
            var azureFunctionSettings = new AzureFunctionSettings();
            config.Bind(azureFunctionSettings);
            builder.Services.AddPnPCore(options =>
            {
                options.DisableTelemetry = true;
                var authProvider = new X509CertificateAuthenticationProvider(azureFunctionSettings.ClientId,
                    azureFunctionSettings.TenantId,
                    StoreName.My,
                    StoreLocation.CurrentUser,
                    azureFunctionSettings.CertificateThumbprint);
                options.DefaultAuthenticationProvider = authProvider;

                options.Sites.Add("Default", new PnP.Core.Services.Builder.Configuration.PnPCoreSiteOptions

                {
                    SiteUrl = azureFunctionSettings.SiteUrl,
                    AuthenticationProvider = authProvider


                });

            });

        }

    }
}

Program.cs

using Microsoft.Azure.Functions.Worker;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Hosting;

var host = new HostBuilder()
    .ConfigureFunctionsWebApplication()
    .ConfigureServices(services =>
    {
        services.AddApplicationInsightsTelemetryWorkerService();
        services.ConfigureFunctionsApplicationInsights();
    })
    .Build();

host.Run();

I am a bit confused, on what will be the durability of the azure durable function? let say my above function to query all the data from a SharePoint list took more than 10 minutes (the limit for an azure function to run in consumption plan),, will the durable function fail? i mean in durable function do we still have the same limit of 10 minutes of durability under consumption plans for each function? if this is the case then what we can do to allow the durable function to run for more than 10 minutes?

Answer 1

Hello @john john Pter - On your specific question about whether the Durable Functions are bound by the 10mins timeout limit when running on Consumption SKU, the following is stated in the docs:

If your function app uses the Consumption plan, you'll still be billed for any time and memory consumed by the abandoned activity function. By default, functions running in the Consumption plan have a timeout of five minutes. If this limit is exceeded, the Azure Functions host is recycled to stop all execution and prevent a runaway billing situation.

Some interesting points come to mind when we unpack the above in the context of your use case.

• Regarding Timeout on Consumption SKU
  • Even though Durable Functions are indeed bound by the 10mins limit, it may not necessarily be a show-stopper for your use case because, in reality, the 10min limit affects the Activity Functions only. When the timeout exceeds and the Functions host is recycled, Orchestrator functions will just Replay leveraging its event sourcing pattern and resume work. This is where the "durability" of Durable Functions comes into the picture: it uses external storage, known as taskHub, to keep track of the work that Durable Functions is carrying out (which in your case is fetching paginated data from Sharepoint), and enables Durable Functions to resume that work if unfinished from sudden failures such as TimeOuts, VM reboots, crashes, etc.
  • You might also look into if waking up the Orchestrator function at a set timer using the DurableOrchestrationContext.CreateTimer method helps too, see: Durable Functions and Consumption plan pricing
  • Fetching 2M records daily in paginated & throttled calls is quite an undertaking in terms of compute and memory utilization, you might consider introducing nested Sub-Orchestrations to your code (as it'll allow distribution of the work across multiple VMs) and coordinating results with design patterns such as Fan Out & Fan in. Perhaps, this is also where choosing Consumption SKU has an upside for you with its dynamic scale relative to variable workload as opposed to ASP and other dedicated SKUs with fixed amount of resources.
• Cost & Billing:
  • The approach described above lets the Functions host recycle & replay the work. If an Activity Function was in progress fetching list data but timeout hits and terminates it, you'll still be billed for any time and memory consumed by the abandoned activity function. I'm not what the exact numbers will be but I wanted to bring Logic Apps to your consideration for the following:
    • Logic app also only runs when it needs to. Plus you pay for actions executed, not for time processing. Logic apps do only have per-execution model pricing.
    • Limits of Logic apps are much more relaxed than functions (for example run duration is 90 days, recurrence time interval up to 500 days …)
    • In logic apps you do not have to write a code – you construct a workflow that has triggers and actions. Sharepoint Connector includes an operation for getting Lists data.

I hope the above helps. Feel free to tag me and comment below if you have any follow-up questions.

---

Please "Accept Answer" if the answer is helpful so that others in the community may benefit from your experience.