Why servers behind loadbalancers can't see their own virtual IP
Many of my customers complain about the fact that when they log into one of their load balanced servers, they can't test to see if virtual IP doing the balancing is doing it's job properly by pointing a browser at it.
This is a universal problem with any layer 3 load balancing device, and is caused by the path the packets take on the return journey. It can best be explained by comparing an external (normal) packet exchange with an internal (broken) one.
The external (working) case:
On recieving a packet with a virtual IP (VIP) as it's destination, the load balancer will change the destination IP in the headers to the real IP address of one of the servers in the load balancing pool and forward it onward. All response packets generated by the server will carry it's real source address, and a destination of the requesting client. The load balancer will then exchange the real source address of these outbound packets for the virtual source address of the VIP and forward the packets outbound to their destination.
The internal (broken) case:
The first half of the exchange works exactly the same as the above. The server sends a packet to the VIP, the load balancer changes the virtual destination address to the real address of one of the servers in the load balancing pool and forwards the packet. This is where things break. The server now sees an incoming packet with it's own real IP address as both source and destination. The server then doesn't bother forwarding the packet out to the load balancer as it considers it purely local (and a waste of time).
A quick and easy hack to get around this is to add manual entries in each machine's /etc/hosts file pointing to the real IP addresses of all the servers involved. This bypasses the load balancer and will provide connectivity between hosts on a load balanced subnet (but won't actually give you any indication of whether or not your load balancing is working as it should).
A not so quick and easy hack to get around this is to use NAT, but this results in all packets passed to a server having the source address of the load balancer itself which is generally a terrible idea from a security/troubleshooting point of view, not to mention a disaster for people who depend on unique impressions of their website to generate income.