It is not a vanity metric.
These points describe a different problem - that you can't condense something complicated like a fund down to a single number, so you shouldn't view IRR in isolation. I suspect (hope) people sticking money into VC funds are sophisticated to know that though.
This isn't unique to IRR. If you stock pick looking only at P/E ratios you're gonna have a bad time too