multiple orchestration instances instead of one

Question

[FunctionName("TestFileOrchestrator")]
    public async Task RunOrchestrator(

        [OrchestrationTrigger] IDurableOrchestrationContext context, ILogger log)

    {

        TestActivityInput input = _TestService.GetNewTestFileDetails(log);

        if( input != null ) 

        { 

            // Batch devices

            int batchSize = int.Parse(deviceBatchSize);

            var batches = _TestService.BatchDevices(input.deviceIds, batchSize);

            //Send C2D message

            var tasks = new List<Task>();

            foreach (var batch in batches)

            {

                log.LogInformation("Adding devices into the batch");

                TestActivityInput batchInput = new TestActivityInput

                {

                    deviceIds = batch,

                    calibMessage = input.calibMessage,

                    calibFileRefId = input.calibFileRefId

                };

                tasks.Add(context.CallActivityAsync("SendC2DMessageBatch", batchInput));

            }

            await Task.WhenAll(tasks);

            //Update the Test lot as released

            _TestService.UpdateTestLotReleaseDetails(input.lstTblTestLotDetails, log);

        }

        else

        {

            log.LogWarning("Test Input data is invalid or empty.");

        }

    }

    [FunctionName("SendC2DMessageBatch")]

    public async Task SendC2DMessageBatch([ActivityTrigger] TestActivityInput input, ILogger log)

    {

        //Send C2D message

        await _TestService.SendTestMessageAsync(input, log);

    }

    [FunctionName("TestTTFunction")]

    public async Task Run(

        [TimerTrigger("%TestFunctionSchedule%")] TimerInfo myTimer,

        [DurableClient] IDurableOrchestrationClient starter,

        ILogger log)

    {

        log.LogInformation("C# Time Trigger TestTTFunction triggerred. ");

        string instanceId = await starter.StartNewAsync("TestFileOrchestrator", null);

        log.LogInformation($"Started orchestration with ID = '{instanceId}'.");

    }

The above code is creting multiple instances of Orchestrator, even though its triggered once.

Note : if something fails inside _TestService.GetNewTestFileDetails(log) , only one instance is created. But whenever the activity functions are triggered (which are running in parallel). then the multiple Orchestration instances are getting created

Answer 1

Hi @VikasRajareddy-9365

Multiple orchestrators in a durable function can start running due to several reasons. One common cause is the replay behavior of orchestrator functions, which use event sourcing to ensure reliable execution and maintain local variable state. If an orchestrator function fails or times out, it will re-execute from the last successful checkpoint, which could lead to multiple orchestrators running if not managed correctly.

Additionally, if the code is not idempotent, meaning it does not produce the same result if executed multiple times, this can also cause multiple orchestrators to start. It’s important to ensure that the orchestrator code is idempotent and to manage the checkpoints and history table storage carefully to prevent multiple instances from running simultaneously.

It's difficult to know exactly what issue you may be running into. In the code you provided, if the TestFileOrchestrator is triggered multiple times, due to the timer trigger or manual invocation, and if the previous instances have not completed their execution, it could lead to multiple instances running. To prevent this, you can check if an instance is already running using the IDurableOrchestrationClient and manage new invocations accordingly. Also, ensure that the TestTTFunction timer trigger is not set to a frequency that overlaps with the duration of the TestFileOrchestrator execution to avoid concurrent runs. When activity functions are triggered and run in parallel, multiple orchestration instances can be created if each activity function invocation leads to a new orchestration instance being started. This can happen if the orchestration client is used within the activity functions to start new orchestrators, which should typically be avoided. I don't believe this is happening, but without knowing what's happening in _TestService.SendTestMessageAsync, I wanted to make this callout. You could be running into concurrence issues with activity functions not being synchronized or incorrect use of external events if they are inadvertently being sent to the orchestrator.

If you haven't done so already, there is a Durable Functions detector under Diagnose and solve problems blade that may provide insights on the execution flow of the orchestration. If not, let me know down below.