You could argue there's an important caveat of symmetric information around this interaction if you want to talk about fairness. Asymmetric information is frequently considered an unfair advantage (see insider trading) but is also recognized as the source of much profit (see auction winner's fallacy), but you have to at least recognize that employers by and large have a huge information advantage in the labor market currently.
It's true that employers are privy to a lot more salary information than employees. I know I said what other people are paid is irrelevant, which I think it is in terms of fairness, but you should be (and, legally at least, in the US, are) free to take that information into account when deciding what you'll accept. It's still up to you to make that decision and hold employers to it. They aren't going to offer up higher salaries out of the kindness of their heart or some sentimentality about fairness.
Unfortunately that's not quite true. Milk (and other foodstuffs) are a requirement to live. People are "willing" to pay more for that because they don't really have many other options. So it's ripe for abuse.