That's not what people who advocate for absolute free speech think. It is an aspect of our broader right to liberty: the right to take such actions as each of us judges proper, so long as we do not initiate force against others. Defamation laws protect are to protect people who feel have been subject to libel speech in a way that affects their reputation or careers for example.

The only reason libel and violent threat are taken seriously is because people go to so much trouble in preventing them. If libel were made legal, people would think harder about the truthfulness of what they read/hear for example.