Why is my scaled Azure Web App scheduled WebJob running multiple instances?

Hello @Rich ,

Thank you for submitting your question on Microsoft Q&A.

Could you please provide the details which I've already shared as private massage.

This issue can be challenging because, while scheduled (triggered) Web Jobs are intended to run on a single instance, there are practical issues such as race conditions, problems with acquiring locks, or deployment quirks that may result in the job running on multiple instances. Here’s an overview of how this is designed to function, reasons you might be seeing this behavior, and some potential workarounds or mitigations.

If you are seeing concurrency errors where both instances attempt the same job, possible reasons include:

1. Race or lock acquisition problems

When two instances start simultaneously, both may try to acquire the lock, and sometimes both succeed due to imperfections in the locking mechanism. Others have reported similar issues with scheduled WebJobs running twice. (Microsoft Learn)

2. Long-running jobs or overlapping schedules

If a job is still running when the next schedule starts, the lock or lease logic can get confused, causing another instance to execute the job. (Microsoft Learn)

3. Settings file issues

If the settings.job file is misplaced, misnamed, or not properly deployed, the scheduler may not recognize the is_singleton or other flags, making the settings ineffective. Stack Overflow

4. Linux, container, or permission issues

On Linux Web Apps, script permissions, shebang lines, and process startup timing can affect how the WebJobs runtime manages locking. Documentation highlights the importance of correct permissions and the existence of the run.* file. Microsoft Learn

5. Stale or broken locks from previous runs

If a previous run left the lock in a bad state, it could cause instances to incorrectly think they can run the job.

6. SDK or Kudu bugs for Linux or concurrency scenarios

There are reports that singleton enforcement may fail for timer-triggered jobs in some setups. (GitHub)

7. Confusion between triggered and continuous modes

The is_singleton setting in Kudu’s settings.job is meant for continuous WebJobs, not triggered ones, so it may be ignored for scheduled jobs. Documentation states this flag is only for continuous jobs. (GitHub)

8. Multiple deployments or startup timing issues

The job might be deployed twice or started by multiple processes, leading to duplication if the WebJobs folder structure causes it to be discovered more than once.

Considering these limitations, I recommend the following strategies to effectively ensure that only one instance executes your scheduled job

Leverage an external scheduling service (recommended)

Instead of relying on WebJob's built-in scheduling and locking, use an external orchestrator or scheduler for precise control. For example:

• Azure Functions with a Timer trigger utilizes blob leases to coordinate across multiple instances, guaranteeing that only one instance runs the scheduled job. (Microsoft Learn)
• Azure Logic Apps, Azure Scheduler (where available), or Azure Automation can trigger your WebJob endpoint or dispatch a queue message.
• Configure Azure Functions (HTTP-triggered) or WebJob (triggered mode) and let an external scheduler start the process.

By adopting this method, you shift the “singleton” responsibility to a robust and managed scheduling service, ensuring reliable job execution.

Implement distributed locking in your code

Enhance your scheduled job by integrating a distributed lock mechanism, such as Azure Blob lease, Azure SQL lock, or Redis. The instance that secures the lock will execute the job, while all others exit immediately. This approach is widely recognized as an effective solution when native locking isn't sufficient.

You can structure your job as follows:

if (!TryAcquireLock()) {

  return;

}

// execute task

ReleaseLock();

With this method, even if multiple instances initiate, only one will perform the intended work.

Implement Singleton with WebJobs SDK

When using the WebJobs SDK, you can ensure singleton execution by applying the [Singleton] or [Singleton(Mode = SingletonMode.Listener)] attribute to your function. This guarantees that only one instance runs across the host, maintaining singleton behavior at the listener level. (Stack Overflow)

Keep in mind, this method is most effective for continuous or triggered WebJobs using the SDK, and may not apply to Kudu-scheduled jobs.

Enforce Singleton with settings.job (Proceed Carefully)

You've already attempted:

{

"is_singleton": true,

"schedule": "0 0 11,16,20 * * 1-5"

}

This approach is valid, but consider:

• Some Kudu documentation indicates is_singleton is not officially supported for triggered/scheduled jobs, but is listed for “Continuous” WebJobs. (GitHub)
• Verify that settings.job is deployed in the correct folder, directly in the WebJob’s root alongside the run script, and that the WebJobs runtime detects it.
• Review Kudu and WebJob logs to confirm the singleton setting is recognized in status messages.

If logs show “WebJob singleton setting is False” or don’t acknowledge is_singleton, the setting isn’t being applied.

Minimize Race Conditions / Ensure Minimum Execution Time

To prevent double execution due to timing issues, you can:

• Add a startup delay, allowing one instance to acquire the lock if two start simultaneously.
• Implement a short locking wait loop before starting main work to secure the lock or lease.
• Make sure the job runs for several seconds to hold the lock long enough, preventing a second instance from mistakenly assuming the lock is free. This workaround is referenced in Microsoft Q&A threads. (Microsoft Learn)

Host the Job on a Dedicated App Service (Single Instance)

For guaranteed singleton execution, deploy the WebJob on a separate App Service configured to run just one instance. This method ensures only one job instance runs, regardless of web app scaling. You can then trigger the job via an endpoint or queue message.

Add Monitoring and Safety Checks

Incorporate guard logic within your job, such as checking a “currently-running” flag in your database. If a second instance starts, it can detect the flag and exit, maintaining safe singleton execution

Here's the plan I recommend for your scenario:

Considering your setup (Linux web app, scaled out, scheduled WebJob):

1. Keep your settings.job configured with is_singleton = true to ensure proper handling.
2. Check Kudu logs to confirm the WebJob runtime is recognizing this setting and logging messages like “singleton lock acquired” or “singleton = true”.
3. Implement a distributed lock in your WebJob logic (such as Azure Blob lease or a SQL-based lock) to provide an extra layer of safety.
4. If reliability is essential, move the scheduling to an external orchestrator like Azure Function timer or Logic App.
5. For added control, consider running your scheduler WebJob on a dedicated, single-instance App Service.

Kindly let us know if the above comment helps or you need further assistance on this issue.

Please "upvote" if the information helped you. This will help us and others in the community as well.