There are certain crucial points you need to be aware of while implementing a load balancer in your work environment.
Web Application and Clock Synchronization
In a load sharing setup, one should not count on the servers clocks for writing data to the database and presenting data to the user. The clocks of servers are skewed. Some servers skew more than the others. Even if a clock is in sync for one day, it doesn't mean that the servers are synchronized.
Application Sessions Type
In case your application uses sessions, it's better to use a database based session. This helps in keeping the user session alive irrespective of which server in your pool the request is being made from.
Server Keep Alive
Using a 'URI keep alive' rather than other methods for example icmp, allows you to remove a server from the pool conveniently by renaming the keep alive file which the Load Balancer is checking for.
Alerts
If feasible, configure email/snmp traps alerts for major incidents, for example server unavailability, service unavailability, etc. The problems are identified in the system through this. It also informs you when the environment status is being degraded
Monitoring
Interface, bandwidth of each service, hits of each service, etc. should be graphed. While investigating for a fault, graphs of previous trends will help you understand the root cause of the problem.
Access lists
The access lists should be setup to restrict the traffic going through the Load balancer. Put explicitly allow access lists on the interfaces and use logging. This helps in identifying the access problems or unwanted traffic generated on the network.