you bigip is setup as follows
Virtual server (load balancer ipaddress)
pool name
host1 - ipaddress of server 1
host2 - ipaddress of server 2
host3 - ipaddress of server 3
host4 - ipaddress of server 4
when you remove say server4 from the pool, the round robin send no more request to server 4
when you add server4 back to the pool, it will get every 4th request. assuming your connections are short lived, the connections should balance.
the behavior you are describing sound like when server4 is added back, it makes a bunch of requests to the virtual server, or triggers some client to it.
you should use a network trace to determine the source of the traffic.