Comment by me-vs-cat | Hacker News Reader

Comment by me-vs-cat | Hacker News Reader

Parent Full thread

I'm taking you at your word, and believe you're stating these things in good faith. As I hope you believe me. Yet I hope what you want never comes to pass. If it does, that world would be confusing, and our society will have lost something valuable.

For example, you say "nobody cares if you reload one page", implying there is a limit -- where is that limit? How many kids in a poor country do I need to hire in order to stay under that per-person limit and continue my business as usual?

> This is not about programs, it's about websites being crawled against the wishes of their authors.

This is very much about programs, because that is the only way anyone accesses a web site. Time-shifting is legal for TV, for example, and that's a good thing, even if broadcasters wish it was banned. We need parallels for the web that balance both users and site owners, and other than robots.txt as a frequently ignored handshake agreement, I don't know much that exists in this area.

Thank you for your condolences. I mentioned the relative passing primarily to explain my past tense and to be honest about not staying updated on that situation despite having been recently informed. They passed from age-related health problems after a good, productive life. I have their picture with a former US president on my wall, from the signing of a disability-related law. I believe my only bias in this area is that I have been directly informed through frank discussion, without the normal barriers you might imagine between non-relatives.

> For example, you say "nobody cares if you reload one page", implying there is a limit -- where is that limit?

You can't demand this degree of specificity from laws - that's not how good laws work. The limit should be reasonableness and intent.

Refreshing hacker news every 5 minutes isn't unreasonable. Refreshing some 5GB file on archive.org that never changes would be unreasonable.

Hell it's easy to come up with examples that would be outright fraud (deceit to acquire something of value) and deserve criminal charges. Refreshing a booking website in a manner to trick it into thinking you're about to reserve something to prevent other people from doing so - e.g. to keep people out of nearby campsites on a camping trip.

> How many kids in a poor country do I need to hire in order to stay under that per-person limit and continue my business as usual?

Up to a jury. But there's clearly a limit at which the answer is "no amount". It's well established that distributed denial of service attacks are illegal after all.

I don't disagree with your overall point. You cannot determine intent by looking at an HTTP request.

Criminalizing a fan who really, really wants Taylor Swift tickets so that you also prevent ticket scalpers, who would do the exact same observable actions, is not going anywhere good.

When limits must be decided like this, our definitions cannot be enforced by technology, and we can't afford judges and juries to handle the scale of the current situation. Look at the original post I replied to, which appears to believe we can easily define "automated" in a way that's useful to meet the wishes of site owners against the wishes of site visitors, regardless of visitor intent and regardless of accommodation for disabled visitors.

> You cannot determine intent by looking at an HTTP request.

I believe this is the crux of our disagreement. Clearly you can't, which is why you must assume good faith and service all innocent-looking requests.

What you can do however is temporarily log and search for patterns in multiple HTTP requests. If you can then prove that these requests came from a bot via the plethora of fingerprinting measures already in use, you have grounds to make a complaint to the FTC, which would have the ability to subpoena the identity of the offending entity and, depending on their history, issue a warning, a fine or file a lawsuit.

The current status quo is that attackers have an advantage in that the worst thing that you can do to them is to temporarily block them, and you don't really want to do this because you're never 100% sure whether any _individual_ request is automated.

While I can't say that this law is good for this purpose, I am pointing out that the solution looks something like it. The goal is not blocking the individual, but doing something about infra-level abusers, at a volume where 95% accuracy is sufficient proof of wrongdoing. All of your arguments are valid and would be tried in court in response to this passing, hopefully for the better but ultimately it depends on how good the justice system is. This last thing is something no amount of technology can fix but in that case illegal scraping is the least of our problems.

> make a complaint to the FTC, which would have the ability to subpoena the identity of the offending entity and, depending on their history, issue a warning, a fine or file a lawsuit.

Good luck with that plan against residential proxies that make one request to your site per IP and don't track requests with any identifying information.

Any propose solution must address that situation.

While I believe we agree on how shitty ad-tech is, there are plenty of ways to screen out outliers from a dataset.

For example, here [1] you stated here that:

> [the crawler operators'] own time to figure out how to get around the block is the biggest cost

This can be proven by applying the countermeasures, then checking which IP addresses (possibly grouped by their most significant bits / AS numbers) have caused a temporary dip in traffic. Rinse and repeat for as many nines as you need to send the motherfuckers to jail.

[1] https://news.ycombinator.com/item?id=50312681

> our definitions cannot be enforced by technology

... that's why the proposed law doesn't try to?

Is there already a limit? If I reload your web page 5000 times per microsecond, am I committing a crime right now under the current law?

Reply on news.ycombinator.com