I don't quite know what you mean by hacker - do you just mean you haven't worked on stuff on the side? As for intimidating - I don't know if I've ever met a hacker in real life that really seemed intimidating. A lot of people develop a sort of persona online, either because they think they're cool or because they're so flooded with work that they simply don't have time to talk to every single person they meet. So the more that you do, the more mail you get from people, and the harder it is to respond to them, and the more it seems like you're exclusive and therefore intimidating.

Don't switch majors because of other people. Switch if you don't like what you're doing. Do you really like programming, exploring code or at the very least making that code work? If so, there's bound to be a place for you. As for "becoming" a hacker: again, try not to look at other people. The path to people having ideas and making projects is always a bit askew and twisted. If you ever want a side project, go ahead and start one. Get your hands dirty. The more you do, the wiser you get. But don't feel like it's a necessity.

Also remember that lots of college students are just learning to code. The self-taught kids have a few years' advantage in a lot of situations. That doesn't necessarily make them better (though a lot more of them have an aptitude for the subject matter). It's certainly possible to get into programming late in your career and still turn it into something meaningful.