I find it scarier when I am the brightest mind in a room, at least on a specific topic. I understand everyone has different backgrounds and experiences, and given the chance to share, many people can make contributions, even sometimes from people you may not expect at first blush.
There are other times though when, not so much through hubris but sort of... delayed understanding(?) - I end up realizing I am the brightest/smartest in a room (at least for the topics at hand). Sometimes you're expected to lead - false modesty won't do anyone much good there. Sometimes you're expected to lead and someone else ends up thinking they're the smartest one, and become a pain to deal with. Looking back I realize I've been that person, and wonder if I could have done anything differently (honestly don't know - lots of factors at play there).