Not really. One could be deaf or blind from birth, yet fully intelligent. It's more a matter of getting sensory from the world into the brain. That sensory may be visual, auditory, touch, etc. The brain is very plastic in this manner.
A good introduction to how the visual cortex works can be found in Eye, Brain, And Vision: http://hubel.med.harvard.edu/book/bcontex.htm
[1] Sorry, don't have a cite, but there are a lot of results on Google talking about it: https://archive.is/iDE27