The typical opposition to speech isn’t that it’s wrong, it’s that it’s dangerous.
It’s correctness may or may not be relevant, but it’s usually the potential consequences of the speech that are creating conflict.
I'm sure the Pope considered Galileo's speech to be a danger, at the very least to the religious dogma of the day.
Sure, there may be clear cut examples when speech is directly dangerous, such as when inciting some group to harm some other group "for no reason".
The problem, though, arises when the situation isn't as clear and when the danger is actually to some people's privilege. To come back to the pope's example: what if some scientist can prove God doesn't exist and / or that the Pope is an impostor? That's clearly dangerous for the Pope. It could be argued that it might even be dangerous for the society, suppose this could generate some unrest. But maybe this could usher in a new era where something else is possible. Should that speech be banned?
The Earth being on an orbit was dangerous.
Non-royalty having legal rights was dangerous.
The prole having legal rights was dangerous.
The idea of electing any public officials was dangerous.
Women having the right to vote was dangerous.
Not invading foreign soil to extract their natural resources was dangerous.
All of these ideas were dangerous and had severe consequences at the time, some challenged scientific norms, some social norms and some effectively undermined the state. You don't get to look back at those ideas with your 2020 hindsight vision and declare they were harmless now because they were radical, revolutionary and terrifyingly dangerous ideas. Why else would the inquisition kill so many people?
How was the argument of legalization of homosexual relations a danger to the establishment's position of power?
If I say that homosexuality is morally wrong and a sin against God, is that "immediately dangerous to the physical person of specific groups"? Does the answer depend on the existence of at least one person deranged enough to take my statement as a call to violence?
If I say that the police are murdering black people in the streets, is that "immediately dangerous to the physical person of specific groups"? If so, is the group black people, or the police?