What is the official language of the company? If it's English, there are grounds to enforce it. The company should _not_ be hiring people on teams who cannot communicate in the common language - that is their failing.
> My manager is remote and has no idea any of this is happening. I don't want to be the asshole to ask people to speak in English, or make my manager be that guy either.
There are several problems here:
1. You can't manage an IRL team remotely. You can manage a remote team remotely, but not an IRL team. Important conversations are happening all the time and managers at least need to be aware they are happening.
This manager should either come into the office or be moved aside for somebody who can.
2. The manager needs to be aware of problems as they happen. They cannot manage what they are not aware of.
3. You are not being an asshole for wanting to communicate with your colleagues. It is _them_ who are deliberately excluding you.
4. (Speaking carefully) Chinese culture specifically can be quite nationalist and diversity of thought and people is not an ideology that is typically shared. If you lost your job and was replaced with another Chinese person, this would be ideal for them.