For one thing, examining referer is a common way that a server determines a request is not a hotlink. Sure you can do something more complicated with cookies or whatever, but lots of sites are just using referer and they'll break if the client doesn't send it.