I just don't think it applies to free speech. And I don't interpret free speech as I can say whatever I want whenever I want, but I am free to express my opinion. This does not include freedom to attack other people's freedom.
Context matters here, I think:
If I said "The president should not be allowed to say this", this would most likely not affect the president's freedom of speech.
If the president said "ls15 should not be allowed to say this", this may more likely affect my ability to express my opinions.