Yes, I would recommend using a true load balancer. DNS Round Robin, while supported, is not a good solution for these reasons.
If you use DNS Round Robin, you may also have to restart clients etc. at times..In your case, restarting Outlook may fix this
Note that others see this too:
https://learn.microsoft.com/en-us/answers/questions/171904/dns-round-robin-for-exchnage-server-2016-cas.html
https://learn.microsoft.com/en-us/exchange/load-balancing-exchange-2013-help