That's not what people who advocate for absolute free speech think. It is an aspect of our broader right to liberty: the right to take such actions as each of us judges proper, so long as we do not initiate force against others. Defamation laws protect are to protect people who feel have been subject to libel speech in a way that affects their reputation or careers for example.