If you don't have confidence in someone's ability based on their experience and their interview but you did like them, give them a task to accomplish offline. See if their results are anything like your results would be, and bring them in to see how they respond to feedback both negative and positive.
I've seen too many interviews that go along the lines of "How would you rate your Java on a scale of 1-5?" "5" "So how would you fix the problem if your cache hit rate on SomeObscureCommercialProduct went from 94% to 82%?" "Forget that guy. Huge ego. Doesn't know anything."
I did run into one company that had an interesting process for technical validation. They actually hire people for two weeks as contractors and have them work with the team. Then they hold a vote and decide whether to extend an offer.