The truth of the matter is that unless you have a very good fundamental knowledge of computer science and primarily math there are things that you cannot tackle as "programmer" unless you are a unique case. A self though programmer is also unlikely to be working on optimizing databases or search engines, compression, driver/os architecture, various algorithms such as video and audio compression and many other things.
I don't claim to be able to do those things and I don't mind thinking of myself as a 21st century mechanic or handyman because in the end I know that my capabilities based on my knowledge and experience are limited.