Welcome to Microsoft Q&A Platform, thanks for posting your query here.
The latency is not related to fewer nodes.
As an example, if your nodepool is spread across AZs, then pods can be spread across AZs increasing the latency but then again, this should be happening even with a much lower set of nodes.
Same reasoning can impact if you’re using internalTrafficPolicy on services, same with much lower nodes.
If your workload is really sensitive to latency, you can try proximity placement groups to reduce latency for Azure Kubernetes Service (AKS) clusters. Refer this document.
Hope this helps.
If the suggested response helped you resolve your issue, please 'Accept as answer', so that it can help others in the community looking for help on similar topics.