Thanks for reaching here! There could be many reasons for this, would suggest you to use the built in Diagnose and Solve blade of your web app, which has the same troubleshooting tools a support engineer would use to assist you.
Recommended steps for troubleshooting High memory issue:
Diagnosing Memory issue in Azure App Service : Part 1 : Identifying the Bad App
Diagnosing Memory issue in Azure App Service : Part 2 : Debugging Memory
This should help tell you if there are any issues with your site that might be impacting performance.
Please let us know if further query or issue remains.