Doesn't Varnish handle many times more concurrent clients than apache does, at significantly lower system load? It could just be pure optimization.
In their "flat-file" setup though - they are deliberately configured so that, should varnish fail (which was one of their concerns) - they don't actually need it - they could just have haproxy immediateley start hitting the apache servers directly.