Yeah, and it's hard to get exposure to areas that might be fertile for inquiry if you aren't spending time with other people who are working on mathematics. It's a social networking effect, in the same way that you pick up programming tricks and get exposure to new frameworks/languages/techniques by spending time with other developers.
I'm sure there is a sizable number of "amateur" mathematicians who work on problems in their free time, but it is hard to know what to work on and where to look.