I do. It's wrong to ignore a robots.txt.
The fact someone else may have licenced content for commercial use doesn't entitle someone else to do the same without paying for it. That's the whole point of copyright law.
The fact someone else may have licenced content for commercial use doesn't entitle someone else to do the same without paying for it. That's the whole point of copyright law.
It's not nice. It is, however, not illegal.
And before you agitate against that, think deeply about the consequences. A world where website owners can legally dictate exactly what software you use to access their content.
And, we already live in a world where website owners can legally dictate exactly what software you use to access their content. They can require that you log in with specific browsers running specific software on your computer, they way online schools run exam proxies.
A website can block Xyz user agent, put Xyz in robots.txt, or even put "No using Xyz!" in a popup and you cannot be prosecuted for using Xyz. It might be hard for you to evade their requirements, but it's not illegal.
Whether or not it's wrong is a matter of opinion. Yours differs from mine, that's fine.
It is wrong to pretend you are deaf to get out of an awkward conversation. No one is going to put cuffs on you for that.
Ignoring robots.txt is not wrong. That's my opinion, just as valid as yours.
It also happens that the legal system agrees with my opinion of the wrongness of this act.
Also, you did say this:
> where website owners can legally dictate
Which is, as I pointed out, false.
I can’t go scrape New York Times articles and make copies of them without first licensing them from NYT.
Unless copyright law has been entirely repealed?
It's one of the reasons why sites that served a lot of data used to have APIs or back channels to get that data even if they technically made money by serving ads on pages of that data, because they were going to get scrapped no matter what, so having a legitimate way to get data that behaved nicely with the servers and was nicely readable by requestors was win-win or at least not lose-lose.