I've met people with much larger vocabularies than myself, and with better grammar, but whose pronunciation was almost insurmountably poor. Someone like me, who has generally good pronunciation, will get praised by Chinese people as having "amazing" pronunciation (and yes, a certain amount of this is just Chinese people being polite but over the years one learns where one stands on this issue) but my comprehension and vocabulary is far and away from fluent yet I will be viewed as having "better" spoken Chinese... never mind that I can only read at an elementary school level.
No matter how much vocabulary you amass and no matter how perfect your grammar not mastering the tones can completely kill any hope you have of Chinese people understanding you. There's nothing worse than the confused look on someone's face after you say something that you think is perfectly comprehensible, only to discover you (ever so slightly, to your ear) screwed up a key tone... it's like suddenly discovering the brakes on your car don't work.