Yeah, I think this is one of those things where if you get yourself into an "Asian languages are hard" mindset, you unnecessarily complicate things. I don't speak Chinese, but I speak Japanese, which I learned primarily from free reading (mostly manga, which is great because it is almost completely conversational Japanese).
While it is true that you can't guess the meaning of characters that you don't know without any context, you definitely can lean the mean of characters that you don't know with context. In the same way, while I might understand the Greek and Roman roots for words in English, if I see the work by itself without any context, I'm not going to be able to understand it. With context, I can usually puzzle it out.
However, it is a huge mistake to ignore the benefits of learning Chinese characters. It is the reverse that is most beneficial. In Chinese the 3600 most common characters cover some insane percentage of the common words in use (only 2200 for Japanese!). These characters form a powerful mnemonic for learning vocabulary. I can easily learn to read the characters for a word and the word faster than I can learn the word by itself. As you learn more characters, it only accelerates your progress.
2-3 thousand characters seems like a daunting task, but as I always point out in these threads, adult level proficiency in a language requires somewhere around 15,000 word families (1 family = a word, all its inflections/conjugations/etc, and all related compound words). People telling you that you can be "fluent" with 2000 words are selling you something (2000 word families is the level of a toddler -- and then people are disappointed that they can only speak like a toddler...).
The absolute best thing you can do for learning a language that has Chinese characters in its writing is to learn those characters. I recommend learning them at the same time as the vocabulary. And to get back to the original comment: even now if I hear an unfamiliar word, I will usually trace what I think the characters are on my hand. Often people will correct me and once I get the correct characters the meaning is almost always obvious.