Hello @Zachary Kingston - AFAIK, with the Consumption SKU with which instances are dynamically added and removed, it's unlikely the front-end load balancer of the HTTP trigger has this issue specific to your unique environment.
With that said, the 20-second delay on Consumption SKU sounds a lot like the Cold Start phenomenon. It can occur after your Functions App has been idle for about ~20min or during scale-to-zero when additional instances will have to be spun up, of which the former seems more consistent with your observation where the 20-second delay occurred even when other hosts exist.
Should your investigation leads more toward Cold Start being the likely root cause, you can try to refactor your function to be more lightweight with minimal dependencies and leverage asynchronous execution wherever possible. For more context, please review the following blog post by the Functions team: Understanding serverless cold start
Moreover, if your function is running a mission-critical requirement and you want to completely eliminate the Cold Start issue, you might consider switching to either the App Service Plan SKU with Always On setting toggled, or even better, Premium SKU through which the Warm-up feature is available.
I hope this is helpful, please let me know if you have any further questions.