@Graham We apologize for the issue you are encountering with seeing growing memory usage within your App Service Plan.
There are two items that you mentioned here. First one is determining why you are seeing an increasing amount of memory usage. The second item is total memory usage per App Service Plan (ASP).
Regarding the first item, there are a lot of variables at play that could be impacting your usage. How many apps do you have on your ASP? How much usage are they using. What app(s) is the overall 20% usage coming from? Do you have any slots? If so, slots are technically their own app and need to be restarted as well to see if any changes are encountered. Lastly, each app has its own SCM site and resources that run with the App. If these supporting resources are not behaving as expected, then yes, something like scaling up/down might help to clear the overall issue. Once you provide an understanding of these issues, we can provide further steps. Note that performance troubleshooting is difficult over forums as we are unable to see your ASPs internal logs to truly understand what is going on.
Secondly, as mentioned above, each App has its own supporting resources. If you have 5 Apps on an ASP, then you have 5 sets of supporting resources also running on the ASP. The more Apps that you have on an ASP, the more "unassociated" memory usage you'll see and not be able to directly link to an app. That is why many on the r/Azure thread were claiming to have discrepancies between their counted memory usage and the reported memory usage.
We look forward to your reply.