Maybe; but if you think like that, then no sample can give you that information. Because you won't get any absolute reference point either way. In any case, such situation is a good reason to talk about your salary.
So your argument is wrong; the sample size is not important (that doesn't mean it's useless, though!). Either you have some reference point, and then you can judge the fairness of the salary (even from one data point), or you don't have a reference point, and then no sample size will help you get it.