Well, because the 13th Amendment by its terms governs private conduct.

I agree, though, that there's a Shelly v. Kraemer type argument that could be made against the enforceability of contracts limiting freedom of speech in certain ways.