PS: This might read as way too broad a statement or as an attack on British people, rest assured that it's not. I genuinely find the British art of social decorum absolutely fascinating and I think it can have tremendous value. Speaking as a Frenchman, I too often witness the cost of being too hot-blooded and blunt in the social rapport.
Are your colleagues mostly Swedish or English/other? Do you speak in English or Swedish to them?
I can imagine if you speak English better than your colleagues then you might imperceptibly/unintentionally treat them like people less intelligent than they are due to having an accent. As a British person working in Europe I've seen it happen, non-UK colleagues I know to be very intelligent are treated ... differently... when talking to someone who speaks english natively. Maybe not "toxic", but in a way I wouldn't appreciate.
I'm joking obviously, but maybe you'd fit in better back home. What's toxic in Sweden might be ambitious in Britain. Cultures are different.