Deleted job still running in Azure App Service WebJob

Question

One month back I deleted a triggered webjob running on Azure App Service & deployed another job with the same name on the same time. However, we observed that the data pushed by the job is duplicated on every run. We tried to contact microsoft support and they gave us link for stack overflow to resolve the matter. But nothing seems to work. I have deleted all the jobs but its still running in the background.

Solutions we already applied:

1. Deleted the web job from the webjob panel.
2. Deleted all the folders and associated files from App_data/Jobs/tirggered/job_name
3. Applied WEBJOBS_STOPPED = 1 as environment variable. (This stopped all the listed jobs but deleted job still running)

Can somebody help me get out of this problem?

Solution applied using below URLs:

https://stackoverflow.com/questions/33790985/azure-webjob-triggered-twice

https://stackoverflow.com/questions/77019665/azure-webjob-running-after-deleted-and-web-app-stopped#:~:text=Sometimes%20even%20after%20deleting%20the,Settings%20to%20stop%20the%20WebJobs.

Answer 1

Hello Raza bin Javed,

Welcome again to the Microsoft Q&A and thank you for posting your questions here.

Problem

I understand that you have deleted a triggered web job from an Azure App Service, but it continues to run in the background, causing data duplication. Despite applying several solutions, including deleting the job from the web job panel, removing associated files, and setting an environment variable to stop the jobs, the issue persists.

Solution

In review the references you provided and what you've done, possible causes of the kind issues here for best practices solution are the followings:

1. The deleted job might still have an orphaned process running due to caching or a misconfiguration.
2. There might be multiple instances of the job running due to a deployment configuration issue.
3. Changes might not be properly synchronized across all instances of the App Service.

So, review the solution you have tried, you can use the link associated with each for more detail steps:

1. Deleting the web job from the web job panel: https://learn.microsoft.com/en-us/azure/app-service/webjobs-create#manage-webjobs
2. Deleting all associated files from App_data/Jobs/triggered/job_name: https://github.com/projectkudu/kudu/wiki/File-Manager
3. Applying WEBJOBS_STOPPED = 1 as an environment variable: https://learn.microsoft.com/en-us/azure/app-service/configure-common

So therefore, if you ensure that your steps are correct. Then, to give permant solution that are best practices for the possible causes of the kind issues as I have explained above. Try each of the options below to solve it, each are independent. Utilize the links for more detail steps if you might require it.

1. Sometimes, a simple restart of the Azure App Service can resolve issues with orphaned processes.
   • Go to your Azure portal.
   • Navigate to the App Service.
   • Click on the "Restart" button.
   • Link: https://learn.microsoft.com/en-us/azure/app-service/quickstart-custom-container?pivots=container-linux#restart-the-app-service
2. Ensure that there are no multiple instances of the App Service running the same job.
   • Go to your Azure portal.
   • Navigate to the App Service Plan.
   • Check the instances running and ensure only one instance is handling the job.
   • Link: https://learn.microsoft.com/en-us/azure/app-service/manage-scale-up
3. If the job is cached, clearing the cache can help.
   • Navigate to the App Service.
   • Go to "Diagnose and solve problems".
   • Search for "Restart and Refresh".
   • Choose to restart the service and clear the cache.
   • Link: https://learn.microsoft.com/en-us/azure/app-service/web-sites-clear-cache
4. Ensure there are no deployment slots that might still have the job running.
   • Go to the App Service.
   • Check the deployment slots.
   • Ensure no slots are running the old job.
   • Link: https://learn.microsoft.com/en-us/azure/app-service/deploy-staging-slots
5. Manually inspect and delete the job files using Kudu.
   • Navigate to your App Service.
   • Go to "Advanced Tools" and click "Go" to open Kudu.
   • Go to D:\home\site\wwwroot\App_Data\jobs\triggered.
   • Ensure all related files and folders are deleted.
   • Link: https://learn.microsoft.com/en-us/azure/app-service/resources-kudu
6. Sometimes, updating the settings or reconfiguring the job can help.
   • Reconfigure the job settings in the Azure portal.
   • Ensure there are no misconfigurations.
   • Link: https://learn.microsoft.com/en-us/azure/app-service/webjobs-sdk-get-started
7. Inspect the resources and ensure no ghost instances of the job are running.
   • Navigate to the Azure Resource Explorer.
   • Inspect the App Service.
   • Ensure no orphaned jobs are listed.
   • Link: https://learn.microsoft.com/en-us/azure/azure-resource-manager/resource-explorer

References

All the links are the best resources used for this solution.

Accept Answer

I hope this is helpful! Do not hesitate to let me know if you have any other questions.

** Please don't forget to close up the thread here by upvoting and accept it as an answer if it is helpful ** so that others in the community facing similar issues can easily find the solution.

Best Regards,

Sina Salam

Answer 2

1. Restart the App Service to clear any residual processes.
2. Check scheduled jobs and the Kudu console for any leftover files or processes.
3. Ensure no continuous WebJobs are running.
4. Review Azure Scheduler and application settings for any triggers.
5. Update the WebJobs SDK and redeploy.
6. Deploy and delete a placeholder job to clear residual state.
7. Review job history for insights.

Answer 3

Hi Sina,

Thanks for the detailed answers. I have checked and verified each step you have mentioned. Please see below notes:

1. We already stopped and restarted the appservice.
2. There are no multiple instances of the webjob rather we have permanently deleted the affected job.

03.Diagnose & Solve Problem page is not showing. It was showing before. So, I couldn't clear the cache.

4. There are no deployment slots associated with the app service.
5. No files or folder exist for mentioned web job in the kudu console.
6. Webjob doesn't exist anymore.
7. Made sure no instance is running on the same app service.

Regarding point 03, I do not know what has happed to Diagnose and Solve problems tab. I think we need to restart the service.