Free speech has many exceptions. So no, you shouldn't be able to link to anything you want to. Just like free speech, there are many exceptions about what you can link to which are already illegal.
By definition, if it's on the internet, it's public facing and therefore publicly available knowledge. Linking is simply telling others, hey, look over here for publicly available information.
Now, if you link to something illegal or perverse, depending on content, that could result in interest from the authorities but that's because you, the user, made a personal decision to directly affiliate yourself with that content via a link. The mere act of linking to publicly available information, should not in and of itself, be illegal or prohibited.