Your experience agrees with my experience of myself. Growing up I was the youngest child in our family friend group and particularly had an older sibling who basically bullied me - I learned to be reticent to speak and was definitely afraid of saying something stupid. When I got a little older (in high school) I eventually got curious about how so many of my peers seemed to converse easily without fear of saying anything stupid. Upon observation I realized that my peers said stupid stuff all the time and for the most part nobody cared. I think to some extent position in the social pecking order probably mattered more for how people would react to what someone said than specifically what they said.

I sometimes think about something similar while watching sitcoms - the characters will often say/do mean things to each other for a laugh and because everyone on the show reacts as though what was said was okay/normal it seems to barely register to me how mean it actually was.

I think growing up with a critical sibling lead to me developing a self-critical voice that I carried around with me so even if externally people didn't react poorly to something "stupid" I said, internally I provided myself that negative feedback that reinforced my fears/beliefs. I think the self-critical voice was protective when it developed - it helped me avoid the more painful external feedback from my sibling, but it continued on long after that was useful.