I think you're spot on with the "they think you're too uptight" hypothesis. Small teams can be really anal when it comes to culture fit, for better or for worse, and unfortunately when it comes to this people can be really prejudiced and ofter rely on more "superficial" factors such as gender and age. I personally work at a startup with a lot of pretty brilliant people, but that has a pretty bro-ey feel at times. I personally enjoy it (I did pledge a frat in college), but I see the term "culture fit" being thrown around when discussing potential candidates and do have mixed feelings about it. On one hand, it does contribute to a sense of community and high morale, and hiring someone that would ruin that sense of community would be pretty bad, but on the other hand, we do end up turning away a few otherwise qualified candidates.
But long story short, they probably didn't hire you because of some shallow first impression (which doesn't automatically make them jerks since we're all guilty of this), but there was probably nothing you could have done so you shouldn't feel bad about it. It's just the way the world works sometimes.