Azure Cosmos DB Serverless, just like other serverless offerings in Azure and elsewhere does not provide SLA's for guaranteed latency or performance. Rather it has an SLO rather of about < 30ms. However, latencies across a WAN can be very unpredictable. Even with provisioned throughput which has SLA's for latency, these only apply to requests made in-region, not across Azure's WAN or public internet and only apply to the .NET or Java SDK's using TCP direct-mode connections.
Another thing to note is that connections take considerable time when initially configuring. If you are making these requests from a cold start from your application, this amount of latency is expected. To avoid this, start your app ahead of time and keep it warm between requests. You can learn more about best practices using Cosmos DB in our docs.