Azure App Service is a service used to create and deploy scalable, mission-critical web apps.
Here I post my solution from the comment section:
"I investigated the issue and found that web app tries to start the Django application before loading the DJANGO_SETTINGS_MODULE variable into os environment causing the URL_CONF parameter not beeing available. As a quick fix, I added DJANGO_SETTINGS_MODULE manually to the environment. Now, all my web app instances work again as expeced. I hope this helps for you too."
Please vote/accept this answer that others can find the proposed solution more easily!