Please see ch4s3's excellent itemized response to an earlier thread for a concise summary of the reasons why this can't be framed simply as an issue of free speech.
The key point is the power differential. While you may be lucky enough to be highly skilled and have enough cultural capital to refuse intrusive questions like this and still find suitable employment on your own terms, many people in our society simply are not in that position. Allowing large powerful corporations to insist that prospective employees disclose their earnings history will simply place downward pressure on wages.
I sympathize with the sentiment, and it is always important to question any limits on speech. But once you factor in the issue of power, pure libertarian ideals will cause greater exploitation of those who have less power to begin with.