I agree, mostly. Being smart and being good at explaining aren't the same thing, though. They may often go together, but it isn't necessarily the case.
One common problem is smart people assuming a certain level of background knowledge, and therefore referencing concepts or using terminology that's not familiar to their audience. It takes some experience to really figure out what vocabulary is appropriate for a particular audience.
I've worked with a lot of fourth graders (10-11 years old). I've seen a lot of people try to explain concepts to them and fail, sometimes because they're using sixth grade vocabulary the kids can't follow, and sometimes because they're using first grade vocabulary and overexplaining things and the kids get bored. It takes practice to explain things at exactly the right level for the audience -- it takes a teacher who is both smart (having mastery of the material) and experienced (knowing what level of complexity is audience-appropriate.)
The assumption of smart people being good at communicating the explanation very well, is not necessarily true. In my personal experience, two of the most smart people, I have come across are basically not the best in communicating. They are so sharp, that thing obvious to them, are not obvious to most of the people.
However, for most people, I find the observation to be correct.