I suppose you are using Azure App Service Linux.
You are right App Service container OS is Debian 9 and it has installed version of GLIBC 2.24.
Unfortunately you cannot customize Azure App Service Docker image, but you can use your own custom image with default GLIBC 2.31 or install any version of GLIBC.
https://learn.microsoft.com/en-us/azure/app-service/quickstart-custom-container?tabs=dotnet&pivots=container-linux
Docker image node:fermium-bullseye has GLIBC 2.31 installed by default
https://hub.docker.com/_/node
Docker image ubuntu:20.04 has GLIBC 2.31 installed by default
https://packages.ubuntu.com/focal/glibc-doc.
Docker image debian:11 has GLIBC 2.31 installed by default
https://packages.debian.org/source/stable/glibc
Dockerfile example:
FROM node:fermium-bullseye
ENV HOST 0.0.0.0
ENV PORT 8080
EXPOSE 8080
ENTRYPOINT ["pm2", "start", "--no-daemon", "/opt/startup/default-static-site.js"]