@Martin Kallukalam Welcome to Microsoft Q&A Forum, Thank you for posting your query here!
We need a deeper investigation on this issue, Need to know exactly is causing problem, it's completely unclear ,There could be several reason for the latency,
I would recommend you to contact support, so If you have a support plan, I request you file a support ticket, else please do let us know, we will try and help you get a one-time free technical support. In this case, could you send an email to AzCommunity[at]Microsoft[dot]com referencing this thread as well as your subscription ID. Please mention "ATTN subm" in the subject field. Thank you for your cooperation on this.
Please do not forget to "Accept the answer” and “up-vote” wherever the information provided helps you, this can be beneficial to other community members.