Scraping Isn't Hacking
getrevue.co
getrevue.co
The problem isn't with the details of any particular scraping or subtly. The problem is with the law, CFAA 1030, and how it can be interpreted however the particular DA wants. The judges are too ignorant of technology to stop them.
Stop with the ageist crap.
I'm not entirely sure what you are saying in response.
Age has positives and negatives associated with it, and ascribing only the latter to it with regard to judges and technology would mean we should set an upper age limit to all court proceedings dealing with any complicated subject - technology is hardly unique in that respect.
There might be an argument for that (or at least some competency test beyond a certain advanced age), but not in the crass, prejudiced way implied in the OP.
I'm familiar with one particular subset of the SCO case. Suffice it to say that a great deal of legal and expert witness time (from both sides) went into creating detailed reports that were intended to be consumed by a non-specialist that laid out history, discussed various claims, and so forth.
A judge could of course choose not to spend the time and effort to digest all this information--but that would be the case whether you're talking technology contract and IP claims or you're talking some complicated and arcane case of real estate contracts.
Lawyers also tend to be specialists to a greater degree than judges. I know quite a few IP/technology licensing/etc. lawyers who are very technically knowledgeable and many of them are not especially young.
But it's what they do for a living. And I'm pretty sure I'll pick one of them to help me out on most technology-related matters rather than a random 30 year old criminal defense lawyer.
Judges tend to be much more generalists. They have to rule on all sorts of arcane points of law which they probably haven't made a career out of studying. But some of them are pretty astute on matters of technology (and at higher levels of the court, they have clerks too.) Look at Judge Alsup in the Oracle-Google case https://www.theverge.com/2017/10/19/16503076/oracle-vs-googl... or Judge Kimball in the SCO case.
Actually, few do. However you can choose your lawyer, and therefore pick one well-versed in the relevant area of technology and who also has a network of expert witnesses and advisors.
"You can't teach an old dog new tricks!"
The average age of newly appointed judges is 50. Yeah, keep on smokin that identity politics shenanigans instead of statistics and science.
I'm not saying a judge should know every field - far from it - but if they could read around just one other subject while in law school and specialise in law surrounding that subject, we may have a hope of having $field literate judges in the future.
Statistically, black people are less likely to have a college degree than white people. Does that mean it is ok to not consider black candidates for hire since statistically, they are less educated? Of course not.
If the problem is that judges don't understand technology, then the solution is to get more judges that do. Not, get younger judges and hope that they understand tech better than their older counterparts because it will probably play out that way statistically.
"Science [in this case, scrapping laws] progresses one funeral at a time".
In reality you see all genders, races and backgrounds. Judges when appointed have a general good balance of genders, races, backgrounda. When voted for represent the voter population makeup. They tend to be older because experience really does help.
>And the rule book on laws regarding the internet is not fully written yet.
Yes, it is certainly true that there's a lot of technology-related law which is pretty unsettled relative to areas of law like contracts and property that have at least centuries of jurisprudence behind them.
ADDED: And one last thing I'd add to the last point is that, when technical people complain about judges (or whoever) not understanding technology when they make some decision, at least some of the time, they understand perfectly well but the existing laws don't lead to the outcome that the technical people want.
Although it sometimes can seem like they do, courts don't usually just make up things out of whole cloth to get the result they personally think is best.
But I've witnessed some particularly aggressive scrapers bring a shopping site to its knees.
I've watched as multiple scrapers were launched at this site from different blocks of IP addresses. I spent some hours blocking these.
After spending even more time, I tracked down these IP blocks - from multiple countries - back to the one scraping company, who had obviously been hired by a competitor to scrape the entire site for its prices for the ranges of goods sold.
I consider such aggressive scraping to be a DDoS attack.
It’s a tough problem though. In an ideal world, I think scrapers should pay for the compute/resources required to serve their requests. I think that’s one of the only ways to scale for this fairly. (This assumes you agree that scraping of public content should be allowed in the first place.)
For example, let's take an API that includes a 'GET /data?start=0&end=100' endpoint that retrieves items with key between 0 and 100.
If items are stored in sorted key order, that could be an efficient and inexpensive query that retrieves exactly the 100 relevant items and doesn't perform any other unnecessary work.
If, on the other hand, the database contains a billion items and they aren't stored in key order, then with a caller-pays pricing model, it's (probably, for the sake of argument) not worth running that query.
In some cases it might be reasonable to continue to pay for the expensive queries. But what if all you need to do is convince the host to 'CREATE INDEX ... ON (key)'?
(all very hypothetical - I don't have any explicit answers. if the host service were open source, the answer might be 'open a pull request and talk to them, and hopefully reduce everyone's costs')
Though, I guess at that point you might as well just sell API access.
In the case at question here the 11th circuit ruled the other way. When circuit courts come to opposite conclusions the Supreme Court will sometimes step in to provide a ruling so that there's unified precedent. Hopefully they'll agree with the 9th circuit.
[1] https://arstechnica.com/tech-policy/2019/09/web-scraping-doe...
Note, I'm talking about the title alone--from the actual article it's obvious they're concerned with legality.
Submit a reasonable amount of requests/second and ramp it up slowly depending on the response time and size/popularity of the website (if it's a national e-commerce brand you can hit it with hundreds per second with no expected ill effects because that's a drop in the bucket compared to their overall traffic, but a very small website on shared hosting might warrant a more cautious approach).
Of course, if they are hostile to it and don't like the "being nice" approach then fight fire with fire and try to get what you want as fast as possible while blending into their overall traffic so they can't identify nor block it. It might sometimes even involve paying real people (on Mechanical Turk or similar) to just browse the website as they normally would and scrape it manually.
The page also outlines how throttling algorithm works, which you use as a starting point for your own work.
Regardless, I’m an ideal world, some polite things would be to:
- ask for permission before scraping, explaining how it’s neutral or positive for their business (so they are more likely to support your continued scraping or even potentially provide a more cost-effective format/API for you)
- scrape at a reasonable pace
- scrape off hours
- make requests with a clear user agent so they know where the requests are coming from
Is relevant here. LinkedIn is invoking the Computer Fraud and Abuse Act for scraping a website without logging in.
While hiQ won a preliminary injunction, the case is ongoing and LinkedIn / Microsoft has asked the Supreme Court to weigh in.
User-agent: scraperbot Disallow: /
And on the topic of trivial workarounds, it's even more trivial to just ignore the robots.txt and scrape away, but that's what malicious bots performing questionable activities do. One would hope a newsbot is being civil and not behaving in a shady manner, but alas, I don't for one moment think any of these news publishers have even heard of a robots.txt
Why does the Supreme Court seem to be considering these two related cases together? seems like duplication?