I've used IPVS a lot in the past and it worked well in a lot of ways. We had a number of problems with ldirectord getting wedged after months of uptime, and the tools are all a little primitive, but IPVS itself works quite well. The amount of traffic it can handle, especially in the "direct routing" configuration, is stunning.
But, I can't imagine going back to it from haproxy. Haproxy allows so many other things including nice status page, agent and service checks, SSL termination, great monitoring (we use Influxdb/telegraf/grafana and Icinga2), and really advanced routing of requests (SNI, path, pretty much anything in the requests).