Web Scraping and Crawling Are Perfectly Legal, Right? (2017)
benbernardblog.com
benbernardblog.com
The law is not as clear cut as people imagine it to be. In many cases the text of the law is less important than precedent and established practice.
In that case a Google without the privacy issues is basically the same as duckduckgo.
Payment and privacy are a very problematic combination, especially where end-to-end encryption is essentially impossible.
You pretty much get to choose one or the other.
One inevitable and useful bias a search engine has is the bias against spam. It's aligned with the interests of the user.
There can be (and are) biases that are against the users' interests but are aligned with operator's: promotion of some content, censorship of other content. This bias I would rather see gone.
It didn't work well enough because back in 2000, Yahoo was one of the first companies to license Google's search engine. (The previous year, AOL licensed Google's search engine.)
Yahoo incorrectly thought that their "portal" and user-submitted "directories" were more important to web surfers than search algorithms like Google's Pagerank. They eventually realized their misjudgment and then tried to build a Google clone through acquisitions like Inktomi and Overture:
https://www.cnet.com/g00/news/yahoo-to-acquire-inktomi/
https://www.cbsnews.com/news/yahoo-buys-search-engine/
https://www.cnet.com/g00/news/yahoo-dumps-google-search-tech...
The price of losing a weak case against a new search engine would be worse for an existing one than avoiding litigation and instead relying on other methods of blocking competition.
The OP who admits he is not a lawyer yet proceeds to advise others. Why? He cannot distinguish the difference between scraping and publishing scraped content or between something being "illegal", i.e. a criminal offense, versus something legal that someone perceives as violating their "rights" and prompts them to file a civil suit. As is typical in the US, such suits may ultimately be meritless and purely used to intimidate.
There are law firms that make money from convincing people to file civil suits against potential competitors. They might list a number of potential causes of action against someone who visits your website. You might be able to sue your users. It does not mean you will win. In losing, you may create legal precedent that in effect enables more of them to violate your "rights".
LinkedIn is taking a calculated risk. If they continue to sue web users who use automation (as does LinkedIn), not every defendant is going be intimidated. Some will defend themselves. I think one of them is going to win.
It is one thing to purport to limit how someone can use some data that you provide to them. That is not "scraping". That is something else. It is another thing to purport to limit how someone can choose to use their own computer, i.e., interactively versus non-interactively (automation).
https://www.vice.com/en_us/article/mgxqb8/the-man-who-broke-...
Even today anyone who wants to opt out of being in the Google (or MSFT, or whoever) search engine can trivially do so by updating their robots.txt. Very very few sites choose to do so.
[1]https://en.wikipedia.org/wiki/Ancillary_copyright_for_press_...
EDIT: Yep. hiQ Labs, Inc. v. LinkedIn Corp., 273 F. Supp. 3d 1099 (N.D. Cal. 2017)
"In summary, the balance of hardships tips sharply in hiQ’s favor. hiQ has demonstrated there are serious questions on the merits. In particular, the Court is doubtful that the Computer Fraud and Abuse Act may be invoked by LinkedIn to punish hiQ for accessing publicly available data; the broad interpretation of the CFAA advocated by LinkedIn, if adopted, could profoundly impact open access to the Internet, a result that Congress could not have intended when it enacted the CFAA over three decades ago. Furthermore, hiQ has raised serious questions as to whether LinkedIn, in blocking hiQ’s access to public data, possibly as a means of limiting competition, violates state law."
I didn't realize original intent could be used in courts. How the heck did "original intent" lead to federal abortion and federal gay marriage when all law in letter and in practice had delegated these questions to the states?
The efforts (and lawsuits) to lock down and control data on the public web threatens to stifle innovation. Websites are only valuable because they're connected to the internet, which for all intents and purposes, is a general utility. It was funded by the U.S. government, and is now managed by a consortium of NGOs such as ICANN, W3C, etc.
When you publish a website, you're distributing it to the world. It's similar to when you publish a book. It has always been the right of those who buy and read a book to be able to use the facts contained therein. Although readers are not allowed to steal the creative elements, they have a basic human right to use the facts however they desire. And the medium doesn't change this truth -- whether you read the book yourself, ask your friend to read it out loud, or employ a bot to read it. So it should be with websites.
Data placed on the public Web is accessible by everyone and should be usable by everyone. It shouldn't matter who parses the page's HTML, whether it's a person, a web browser, a bot or a robot.
We're starting a movement to ensure that the facts of the world are always available for innovators to build on top of:
It's similar to when you publish a book.
So just don't accept the ToS. They can't make you accept it, no matter what "by [breathing] you accept our ToS" verbiage they put in it.
Sure, maybe that would make your access of the website "hacking" ("accessing a computer without permission", since you rejected the contract which would grant permission)... but the court rejected that argument in HiQ vs LinkedIn.
[I get that this probably isn't how a typical court would see it, but to me that reflects pro-corporate bias and corruption more than any kind of sane application of common law.]
[1]. Edit: originally this read "charged", but "sued" is more accurate. But I guess what I really mean is "successfully sued". Anyhow.
Imagine that in the real world. I honestly expect it to become commonplace - by entering our store, you agree to the terms posted in the binder you may ask to view.
For a specific example, it is legal to conceal carry in public spaces in my state, but I have seen a lot of stores having some extra rules posted on their window saying that by entering the store you agree to their no-carry rule, among others.
If the property owner posts a sign that says no guns, and you carry anyway, you're guilty of an offence.
Source: https://statutes.capitol.texas.gov/Docs/PE/htm/PE.30.htm#30....
>Imagine that in the real world. I honestly expect it to become commonplace - by entering our store, you agree to the terms posted in the binder you may ask to view.
My point to this was that it isn't that much of a far-fetched scenario, as it already happens with "no guns on store premises" signs in my state. You consent to the arbitrary rule by entering the store. I bet it is already backed up by state law (not sure 100%, but I believe so), just like it is in Texas (I am in WA), so I agree with you.
By your logic, if someone shoots a bullet and it kills someone else, it would be the bullet's fault and not the person who pulled the trigger...?
If someone uses your open source software to commit a crime, is that your fault?
Now legally if you can prove that the bot author is aware of the TOS, that's a different situation, but there are a variety of situations where it is completely reasonable (search engine bots in particular) where the author will never know about the TOS.
If I write a script that goes to every web page on the internet and fills in all text fields with "jdJdkjl28d", selects a random choice from all drop-downs, checks all check boxes, and hits submit, no one is going to seriously argue that I would be bound by any of those agreements because I didn't read them, and the logic I wrote isn't even aware that they are agreements -- it is just checking checkboxes regardless of what the form says.
This is why some companies make you scroll all the way down to "read" the agreement before you can check the box because then it can be argued that you actually read the agreement.
> "[W]here a website makes its terms of use available via a conspicuous hyperlink on every page of the website but otherwise provides no notice to users nor prompts them to take any affirmative action to demonstrate assent, even close proximity of the hyperlink to relevant buttons users must click on – without more – is insufficient to give rise to constructive notice."
The easy example are teen curfews at malls. If you look under 18, you consent to providing proof of age to enter certain areas of the mall, and cannot do so without an adult over a certain age.
They cannot however put a "If you're under 18 and violate the curfew, you hereby agree to pay a contractual penalty of $1000" in their house rules and sue you for that money even tho you did not consent to their contract.
Moreover, it would fail the test of acceptance, because the teen in question would have had to have known about the contract and consented to it, neither of which the mall could likely demonstrate, unless e.g. they could prove they explicitly showed the thing to the teen before entering, not just hung it somewhere.
And lastly, the teen being a minor, the contract would be void in most jurisdictions anyway as teens lack the capacity to enter contracts in most jurisdictions.
If the teen caused actual damaged, then tort law is still an option, of course. And if the teen refuses to leave or sneaks back in later, then we're talking criminal offenses.
In the Linked-In case here, someone didn’t just happen to involuntarily agree to the TOS by making an HTTP request. They agreed to the TOS as part of creating a new account. That’s an affirmative act, as part of a bargained-for exchange. It’s no more “coercive,” in substance or form, than the release I signed the other day to rent a jet ski. You want the benefit of access? Abide by the terms.
After having agreed to the terms, the Does violated them repeatedly on a mass scale. Again, this wasn’t accidental. The complaint lists a litany of ways where the scrapers clearly encountered technical barriers Linked In put in place, and used technical means to get around them.
In that case, can people get in trouble for scraping such HTTP resources at scale, all around the Internet?
I am worried not about breaching the TOS, but the kind of "hacking" definition that involves "circumventing the intended use" regardless of technological availability and never agreeing to a single contract.
To go back to the Linked-In example of the article. The predicate for the CFAA claim is not merely scraping. It’s intentionally bypassing technological measures intended to block the scraping, after there was notice that the scrapping was not permitted.
It’s very much like real life. If a shop keeper has the door open, he can’t sue you for trespass for walking in and looking around. That’s called implied license. But if he kicks you out and tells you not to come back, you can be prosecuted for trespass if you come back, even if the shop keeper leaves the door open for everyone else. An open web server is similar.
You generally can't even _see_ the ToS without making further requests.
Something along the lines "by using this site, you agree to our terms of service and use of cookies". You'll definitely see a banner like this on paywalled news sites.
Thing is, I got to the site from a link from say HN. I haven't agreed to anything, let alone seen the ToS, let alone had a chance to affirm or unaffirm to the conditions until I've already used the website. Thus, they assume my tacit approval of their ToS. The only way to not agree to the ToS is to never have visited the site to begin with. Yet, you cannot know the ToS unless you visit the site, and even if you knew the exact link to their ToS, that is still construed as usage, and once again assumed implicit approval.
Additionally, if I'm accessing publicly available material without being authenticated, you cannot prove I ever agreed to your ToS.
I don't work on scrapers or anything, but I abhor the idea that I have implicitly entered into a contract by virtue of visiting a public, unauthenticated, webpage.
Want me into a contractual ToS? Make me sign up and sign in.
This is something I have seen getting a lot of rise since the introduction of the GDPR on German/European sites.
The law text of the GDPR on the other hand is kind of strict in that you really have to agree to something by clicking and this "accept by doing nothing"-method is explicitly not valid.
Curious to see when the first people will go to court against this.
Also, they aren't 'forcing' you to accept their TOS. You are free to stop requesting from their site.
Alternatively, how can any website prove the TOS was even mentioned at the time a request was made? You can't enforce a contract that wasn't available to be consented to. You gonna put a TOS version at the head of every website, and have that version archived somewhere accessible later?
Edit: This is also why you should never evade a ban. It's potentially a federal felony.
[1] https://en.wikipedia.org/wiki/Computer_Fraud_and_Abuse_Act
In most cases described as "gray area" it comes down to cases where the stronger party made a compelling argument against what might have been a very valid reason to scrape. The threat is often wielded as intimidation and intimidation alone.
via https://en.wikipedia.org/wiki/American_rule_(attorney%27s_fe...
While you rarely, if ever, see cases where losing parties have to pay millions - I personally haven't seen any like that - you can see people having to pay tens to lower hundred thousand $ in legal fees.
That can be a substantial sum for regular people, and those circumstances happen when the other party has hired in high-$$$ lawyers, and the case goes on for a while. Especially if there's multiple appeals.
So there's always the risk of that.
If you want to sue a big company for treating you poorly, you risk a very high price of you lose.
Unusually high medical bills in the US are also probably a contributing factor. I’d be curious to see how often a civil suit is brought after a car accident in EU vs US.
Depends on the country of course (UK is probably closer to USA instead of the rest of EU with civil law), but in general that would very rarely result in a dispute that needs to be resolved in court. The mandatory liability insurance would cover the usual claims without a civil suit, and unusual claims are unlikely to succeed and expensive to pursue, so they aren't.
Vis the derogatory term "ambulance chasers" for personal injury lawyers.
You've just made a container image that does something cool, and have published it on your site, which is running on the cheapest Amazon Lightsail plan (in Ohio). The final container image is 50 MB in size.
(Yes, it's weird that you wouldn't publish it to Docker Hub or the like. Suspension of disbelief, please…)
A grad student somewhere decides they like your container. They write a script that downloads your container to $TMP, and runs it with their input. Over a week, the student runs their job 50,000 times.
(Running a compute job 50,000 times is not unusual in HPC. Each run has a different input, and since your cluster would have a job scheduler, breaking up the work helps make better use of the cluster.)
Because the student downloaded the container to $TMP, a new download has to happen each time the job is run. 50,000 downloads of a 50 MB container is 2.5 TB.
(The docs say TB and GB, so I'm assuming base-10 instead of base-2 here.)
Your lightsail plan is $3.50 per month, and includes 1 TB of transfer. You will have at least 1.5 TB of extra transfer this month. At 9¢ per GB, 1500 GB of extra transfer is $135.
Please do not dismiss this out-of-hand. The scenario may seem a bit contrived, but unexpected bandwidth charges do hit people, and can make you really question your dedication, when having to pay for the access by people you don't know.
The actual number of lines is not important
This begs the questions;
- At what point does web-scraping become research?
- Is it considered scraping if a human is actioning it (no automation)?
- Could I manually click links and copy the data from the browser, is this scraping?
There's no strict definition, so it's up to a judges and lawyers to decide what the definitions entail...
Not only is it up to judges and lawyers but there's >190 juresdictions in which you could be sued. There are more important things to worry about, such as heart disease.
Exactly, so much of humanity is wasting time needlessly arguing in courts... we need better systems of laws to reduce these redundant debates...
Making automated requests isn't illegal and won't per se get you sued, it's literally how a web browser works. What gets you sued is pissing somebody off. So don't do that.
Seriously, if one expects some privacy, they don’t live in a glass house and then complain if you see ‘em, no matter what ToS on the entrance claims.
For example on the issue of insta-agree TOS's that you are in no way forced to read ("by being on this website you agree to our TOS" BS), is it possible to make cases to establish precedence from that? Let's say I set up a site where the TOS says everyone that views the site agrees to give me $5 and I sue someone -- clearly that wouldn't hold up in court (gulp, hopefully) -- and having it struck down would help clear up this gray area...
In the US, the law has kind of settle in favor of scrapers. Thanks to the first amendment and the fair use exemption. It the data doesn’t require to sign up and your are not DDOS the website. Europe is another story.
1) How liable is the author of a scraper who put it on GitHub but never did use it themselves. Because I remember seeing a lot of implementations and tutorials of tools like BeautifulSoup that don't necessarily comply with the ToS.
2) What if you edit the User-Agent to something that resembles a browser provided the scraper is just substituting browser interaction, e.g. those search from your terminal applications. Could/would it be construed as impersonation or something?
China’s Draft Data Security Measures and How They Compare to the GDPR: https://www.natlawreview.com/article/china-s-draft-data-secu...
2. Add a ToS notice stating that anyone downloading it now owes you $1000.
3. List every client IP in the logs and sue everyone even if you don't know who they belong to.
4. Profit?
https://pando.com/2014/02/06/who-owns-real-time-sports-data/