@Tomasz Kornacki Based on what you described, it appears that the Azure SQL database is deployed as a serverless compute tier. The database is automatically paused during inactive periods and automatically resumes databases when activity returns.
The delay you are experiencing in the first request is the "warm up" time when the database is being resumed.
Screenshot below for reference. I have mine configured to Auto-Pause if the database doesn't have any activity for 1 hour.
If this is not the issue in your case, please let us know and we will troubleshoot further.
If an answer is helpful, please "Accept answer" or "Up-Vote" for the same which might be beneficial to other community members reading this thread.