I'm assuming you're using multi-tenant and not an App Service Environment. Outbound IP addresses on multi-tenant environments aren't static and subject change. For this reason, it's always best to use the FQDN, as @ShamirC suggests. This includes binding a custom domain using a CNAME record. However, if you're still wanting access through IP, one method is using IP-based SSL binding which Secure a custom DNS Record with TLS/SSL binding | Microsoft Learn walks your through. Another option is to integrate your app service with a NAT gateway.
These, and any other options, all additional overheard. So, I wouldn't advise going that route unless there's a strange reason why the FQDN wouldn't work.