Hacker News Reader

Comment by bradley13 | Hacker News Reader

Anecdote: For my programming classes, one example I use is a simple browser. It doesn't do CSS or Javacript, so display is primitive, but it works.

On some sites. Many sites, especially the big ones, see that it's an unknown browser, and refuse to send content. Probably they think it's a bot. But even if it were, what's wrong with bots, as long as they're well-behaved?

What kind of closed web have we let the megacorps build?

View on HN

I mean, the site gets to decide if it will service a request. theres no requirement for a service to respond to everyone.

This is the tech equivalent of "we don't serve your kind here", and limits any attempt at competition.

Playing the devil's advocate: Why shouldn't a server get to decide which clients it wants to talk to?

Why shouldn't a store get to decide the $protected_class of which customers it will do business with?

User Agent is not a protected class. Neither is Intentionally Obtuse for that matter.

In principle? Of course. I mean I remember blocking Yandex bots from hammering some e-commerce site on shoestring budget.

But each person developing a web scrapping bot realizes at most after a week that being honest with User-agent has negative impact on how well it works, and changing it to existing browser takes literally seconds.

Long term, I can imagine this becoming a corollary to Net Neutrality. Whenever that finally becomes a thing, the next step could be a law/rule that "public web sites need to be accessible by the public". Not that developers need to test their work in dozens of different browsers, but that they can't actively choose their customers.

Reply on news.ycombinator.com