Restricting "freedom of speech" to simply the ability to function in an environment with no intrapersonal or governmental violence gives us at best a faded facsimile of it.

Most "totalitarian" countries don't function by literally throwing people in prison or beating them for saying Bad Things: that's far too expensive to be total and widespread. It functions by removing people from positions of media exposure; by denying them access to physical or network media; by ascribing implicit or explicit social credit scores that governs access to resources. None of those things are violent and rarely even explicitly ordered by the State. In fact, "no one has the right to a platform to hurt society/the nation" is how those totalitarian regimes typically justify their repression. And they're still every bit as much a restriction on freedom of speech.