That's the big problem, isn't it? People think it's okay to give out information on an incoming call because often it is really okay. If it were unreliable 99% of the time, phishers would not use this method as an attack vector.