Making the wrong call would entail the interviewee potentially feeling offended, which could be an issue if the ultimate decision was to make an offer to them -- they might reject the offer due to this experience alone. Not to mention it would be pretty awkward to work with a new colleague where you walked out on him as an interviewer. (In general you really don't want colleagues to know that you voted to pass on them)
In general if a candidate thinks they're getting nowhere they can suggest to stop the interview. Nobody can stop you from cutting it short and leave if those 50 minutes are important to you.
(Also, FWIW I don't think saving a hypothetical 50 minutes of interview time is what the GP was specifically referring to...)