We managed to do it actually with the following way
- Created a virtual network
- Configured storage's firewall to allow access only from the virtual network
- Created an application gateway (instead of a front door) on the same virtual network that routes all traffic from a public ip to the url of the static website hosted in the blob storage.
With this setup we are able to access the Static Website through the public ip of the gateway but we are not able to access the website directly from its own url.