Building custom solutions that extend, automate, and integrate Microsoft 365 apps.
That's a very good observation also a great question!
Yes, if there has been no script execution for a while, and a new execution comes in again, this new execution will experience a little latency. The succeeding executions should be faster afterwards.
This is to avoid the cloud compute from being "idling" for too long so that we can fully utilize the available resources to serve other customers.
Also as a cloud solution, there are definitely caching in different parts of the service. This also means normally the first execution (in a while) would have to pay that performance cost for succeeding executions.
So if it's possible, I think it'd be best for you to skip the first execution in your performance tests.