I would imagine it’s analogous to blocking phone numbers on the White House switchboard: freedom of speech in this context means that the President cannot artificially limit who speaks to him through publicly-available means.