Well, if the load balancer is down, then yes you are dead until you fix it :)
The load balancer is a single name space that all the clients can connect to and you can mark down a server down manually or the load balancer will mark it down if its not available:
Managed Availability includes an offline responder. When the offline responder is invoked, the affected protocol (or server) is removed from service.
To ensure that load balancers do not route traffic to a Mailbox server that Managed Availability has marked as offline, load balancer health probes must be configured to check <virtualdirectory>/healthcheck.htm, for example, <https://mail.contoso.com/owa/healthcheck.htm>.
I recommend you read through the link I posted. If you decide you dont need a load balancer, then you will see the issues you describe and clients may lose connectivity to Exchange when one server is down. There are many options out there for 3rd party load balancers: F5, Kemp etc...
You can even use round robin DNS:
Third party vendors
Example:
https://kemptechnologies.com/microsoft-load-balancing/load-balancing-microsoft-exchange