Hi,
How do you access sites which are hosted in Application gateway ? Is it via the Traffic Manager mapped URL ?
How are you determining that the second site in your AppGW is taking 5 mins ? Did you tried accessing the site using AppGW's public IP directly or are you using traffic manager's URL ?
If you are using traffic manager mapped URL, then you will need to reduce the TTL setting on your Traffic Manger profile, so the client's DNS queries are not cached.
Regards,
Karthik Srinivas