This is true, and unsurprising given the historical context. In fact, the Constitution was written as such with the original intention of providing those freedoms primarily to the benefit of not just white men, but land-owning white men. Crofters, smallholders, and other types of land-owners wanted to be put on the same tier of political power as others, when previously they were stifled under the aristocracy. That was the defining feature of the American Revolution and the things which followed.
But your reply to me is primarily just a form of whataboutism and fails to refute or respond to my core point. My view of history is neither narrow nor incomplete, insofar as it applies to the racial and gender injustices that existed in the US. This also has pretty much nothing to do with the philosophical basis of free speech.