Additionally, -If your requirement fits, you could look at ILB ASE. The deployment of the App Service Environment with an internal endpoint is called an internal load balancer (ILB) App Service environment (ASE).
If you haven't done, you may review logs/metrics to narrow the issue further - Troubleshoot using metrics - metrics that can indicate issues either at Application Gateway or the backend.
--- Review the architectures Access-restriction-rules-based-on-service-endpoints and leverage as per your needs.
- Configure App Service with Application Gateway and integrate-with-application-gateway for config settings.