You doubt that people can talk objectively and confidently about this, but then it seems like you conclude (implicitly or not) that we should therefore stick to the default of assuming everyone's experience is the same (and therefore the same as yours, btw). Alternatively you could take the difficulty of communication as a sign that we have at least just that much leeway concerning the difference between each others' experience.

You are skeptical when people say they have an internal monologue. Those without internal monologue would be skeptical when you say you do have an internal monologue. Your confidence is also based on nothing more than your personal experience + bias.

PS I firmly doubt those "swapped colour experience" ideas make any sense at all. Probably the thing that is assumed to be flipped is just that part of the experience that only seems to exist as an absolute, yet is actually not defined at all beyond its relations to other experiences.