It's the startup time of your app. When your app is not running and then receives its first request then it needs to start up and run through your startup process. You should notice a similar (but probably not as long) startup time when running locally. Of course the less you do at startup the faster the app will run. You can profile your app locally to help see if there are any major issues. EF, for example, is pretty expensive to start up when you hit the DB the first time.
Upgrading your app service plan would help a little. Specifically if you look at the available service plans you can see that the free tier in on a shared VM. Therefore you are sharing time with other apps. If other apps are busy then your app is going to run slow as well. But it is free so you get what you pay for. You can upgrade to a better service plan including a dedicated one for more money. This will give you better overall performance but if your app is only used periodically you'll be throwing money away. You need to determine which option is best for your needs. Besides how much it'll run you should examine how CPU and memory intensive it is. Higher tiers have a better VM backing them.
On the DB side, if you notice your queries are running slow then upgrading the DB would be an option as well. On the pricing page you can determine which plan works best for you. If the issue is only at startup then changing the DB plan is probably not going to help.
If you're using EF then it has to compile your data model which can be expensive. EF Core 6 will have compiled models to help speed this up dramatically but until then your best option is to use bounding contexts to separate your EF models into only what you need. But it sounds like you have a pretty simple app and therefore your DB is probably not that complex either so making changes to EF at this point won't help performance.