"Hands-on" skills are not the only valuable skill (however you define them), and being able to communicate effectively is sometimes more important. With today's prerogative of building "things" (whether that's a product or a research paper) as a team, communication skills easily slide to the top.
What you may perceive as "charisma" is probably a better communication skill: it can be learned as well.
Now notably, academic circles are generally festered with politics, and it takes a really strong will to let your skill do the talking. But in commercial settings, good managers are aware that you can get away with a few bad communicators if they are extraordinarily skilled, as long as you channel their energy properly.
In general, you can rarely control what others do, so the best starting point is to control what you do: try to train yourself to manage your emotions, to be more diplomatic in how you critique others (perhaps check out Radical Candor?), and how to gain the skill to let your skills shine through without getting pulled down to the "gossip" level.
Sometimes easier said than done, when the best way out is, well... out.