At small, purely algo shops (ala Jane Street), the traders are your nerds. They're programmers, engineers, mathmeticians, physicists, but they just happen to trade.
The incredibly aggressive people that you're talking about is mostly floor traders, brokers, etc, because they need to be that way, and they're generally on their way out, as algos will be smarter and faster.
I've worked in a variety of industries (in software development), and am currently at an HFT shop, and I've found this to be the first place I feel actually values my abilities.