Your opinion might change after someone sends messages to your contacts as you by spoofing your phone number (very easy).
Friends complain about falling victim to Facebook scams all the time but I have never heard a complaint about WhatsApp … I know that such an anecdote is not of much use but WhatsApp seems at least not to have lost the trust of its users so far.
But I (personally) consider the probability of it happening x potential damage to be insignificant when compared with convenience offered.