Why is there a slow response time when js bundle is served for the first time in Azure?

Question

My application runs on asp.net core as backend and angular 9 in front end. I have hosted my site using Azure App Service. I am facing this weird issue of slow overall response time when a static asset is served from azure for the first time. When the js bundles are downloaded in the first request, and when I clear the cache and revisit the same page to fetch the bundles, I don't face any such issue and the bundles are downloaded without much delay.

When I test this same app locally, i.e when I run this app in IIS express, I don't face any such issue.

One thing to note is when I "Emty cache and hard reload" the app, the issue gets replicated again only in when looking in the app hosted in Azure.

The cultprits here seems to be TTFB (Time to First Byte) but if its true then why does it not affect the response in the second refresh? Also, sometimes the issue seems to be slow download time where as the size of the bundle is ~15 kb.

Here are some screenshots of the issue:

First visit(notice the size and download time): first visit and ttfb image and overall view from first visit

Second visit: second visit

I tried writing this in web.config:

<serverRuntime enabled="true" frequentHitThreshold="1" frequentHitTimePeriod="00:00:20" /> that I found from here but the issue still persists.