A good reputation with solid references that are checked from real engineers and managers that you actually delivered a good product seems like it would suffice to override the straight up lying, to me. Is that not the case?
The reference knows the candidate and likely feels a much closer connection with them than they do with you. Assuming the candidate isn't an asshole, they will want to help them out by giving them a good recommendation. If the candidate was let go due to downsizing or such, the reference may even feel pressure to help the candidate find a new job.
So it's a great data point to have and as you say, it can help weed out the straight up liars.
But any 2 given candidates with 5 years at Acme Inc are not interchangeable. Some candidates are just better at programming (or some subset of it) than others, and I think employers try to find ways to predict that at interviews.