@Hannes Caesar since App Service is PaaS and runs in a sandbox environment, I don't believe glibc can be upgraded.
You can try using PRE_BUILD_SCRIPT_PATH to set a script file that attempt to update/upgrade the glibc. I haven't personally tried but if requires system elevated privileges, it more than likely won't work. The best workaround though is using a custom container and configure your image with the Debian version you need.
Furthermore, WEBSITE_LINUX_OS_VERSION refers to LinxuFxVersion so acceptable values are az webapp list-runtimes --linux.