Show HN: CloudScrape – Cloud-based web scraping platform
cloudscrape.com
cloudscrape.com
The tooling and intuitivity is awesome but besides that at the heart doesn't it do what any other headless javascript enabled browser does?
For example: https://phantomjscloud.com/site/pricing.html
Is there a way to get a less feature rich version of this i.e. sans auto-resolve CAPTCHA etc?
Great work none the less though.
We see a lot of our clients manage to do what they need done - even daily scrapes - for as little as $29 / month since scraping a news site daily will often take up no more than a few minutes.
FYI, I have a public preview of our new version, totally free. Check it out if you find this kind of service interesting! http://stage2.phantomjscloud.com
Downsides were like so many other crawlers/spiders - too much in trying to meet the needs of all. Aiming for the 95% is easy, 95 to 99% though, not so much. Hence all the corpses littered along this path.
Recommendations:
Define solid, real-world examples of your platform in practice and really, no bullshit, what makes you different from every other platform that's 97% polished out there?
You didn't meet my needs because I needed much more in the way of parsing network requests from pages, mostly in handling if/then scenarios. A relatively simple scenario, to ME, and there-in lies the challenge of the crawler market, it's the last 5% that's custom to each scenario that so often delivers 98& of the value.
Suggestion: Identify and use the talent you obviously have to destroy some high value markets. Hint: Prioritize targets that still champion a desktop app at any point in the top 5 in terms of data collection.
Outside of that and far more niche (and subsequently far lower priority, allow for parsing / handling handling network requests and responses via custom app/API instead of just block all, some, or none)
Just because you have access to something doesn't give you permission to access or access it in any manner possible.
Nobody is taking anything. If you don't want someone to access your page, then don't respond to their request.
Most scrapers - including this one - advertise how they use multiple servers/locations/ips/etc to get around this.
Even if identification was hard, which is not true because of how HTTP works, it is irrelevant because HTTP doesn't discriminate. If someone does, that is their problem, and should be solved by them, and not a committee or law.
> there's no easy way to always reliably identify the requester
That's the problem: you can't identify the person to block them in the first place.
Robots.txt is actually an explicit signal of intention for reputable search engines but that's all we have today and is easily ignored and does not work with these scrapers or anyone else.
Not sure what your last sentence means.
When you say "That is not how HTTP works" it suggests that your claim is that anything that HTTP allows is ethically OK to do. However that is clearly a ridiculous stance, since a DDoS attack is a stream of valid HTTP requests and that's clearly not OK.
So I'm left wondering what your argument actually is for why unwelcome scraping is ethically OK.
I find this an interesting question, because while I would love for protcols to also define ethics, I feel that would be scope creep for the poor protocol designers. There's a wide variety of conduct and ethics questions that a protocol cannot address.
Where I myself draw the line is at protocol behavior intentionally designed to obscure my intentions. For example, sending my requests from a wide variety of IP addresses is behavior that is specifically designed to obscure where I'm coming from; my only intent in doing so would be to circumvent the intent of the serving machine from providing lots of content to a single requestor. At that point I'm engaging in deceptive behavior; I've crossed an ethical line.
That wasn't a response made to your comment, and you are mixing two different arguments there. You guess in not correct.
So I'm left wondering what your argument actually is for why unwelcome scraping is ethically OK.
I never even suggested such an argument.
The behavior you described in the last paragraph is only deceptive from the eyes of an information and privacy surveillant state actor. Anonymity is not unethical, it is a human right.
Services like this one actually make scraper-related unavailability, which IMO is already greatly exaggerated, less likely, since there will be fewer amateurs trying to write their own bots and accidentally breaking things.
To the extent that a scraper harms the other business, the scraping company can be held civilly liable on several accounts without specifically bringing scraping as a practice into the picture. All that matters is that they damaged the target site's ability to operate, not that they were saving [portions of] the pages (that'd be a separate copyright claim, unrelated to the disruption of service).
Web is for people from people, not solely for company(s) financial interests.
Intentions matter - on both sides. This is what most of the legal framework of the entire world is based on. You can disagree with that but again the ability to do something doesn't grant permission to do it. You're saying the solution to that is to remove the ability, but I don't see how that's realistic.
What it has to do with it is that putting an HTTP server on the public Web signals the intention to serve up resources to anyone who sends an HTTP request. Any restrictions to this default must be implemented explicitly on top of the default.
Leaving my stuff lying around does not signal my intention for anyone to take it, unless I let it lying around next to the bins.
So yes, intentions matter. The question is how we learn about them. Sometimes the choice of technology implies particular intentions by default.
We already have an explicit signal called robots.txt which major search engines use. The problem is that there's no way to enforce this and there's just very little enforcement against actions on the web in general which is why people can get away with scraping but please don't mistake it for somehow being OK or allowed by the owner of that content. It's just not that simple.
This is legally incorrect. Without any further information or protection measures by the publisher it is legal to access content on a public web server.
">I really can't say this in any other way - just because you can do something doesn't mean you are allowed to"
You are allowed to do everything that is not expressly forbidden by law. Accessing a public web server is not forbidden by law unless the owner takes steps to prevent you from accessing it or at least clearly signals that intention. Terms of service do not constitute an implied contract, so you are not required to read the TOS before accessing a public page.
">We already have an explicit signal called robots.txt"
Exactly. That is part of what I meant when I said that any restrictions had to be implemented on top of the default, which is that everyone who can is allowed to access anything on a public webserver.
[Edit] But initially, this wasn't a thread about legality but about ethics. I think there are unethical reasons to scrape and there are unethical reasons to block scrapers. We simply need to know more about the purpose of any scraping before making a judgement.
2) This directly contradicts your previous comment: "Leaving my stuff lying around does not signal my intention for anyone to take it, unless I let it lying around next to the bins."
3) There is already case law precedent regarding this exact type of publically accessible information not being authorized: http://www.net-security.org/secworld.php?id=14614
I agree that this is about ethics and any scraper that doesnt honor robots.txt and explicitly uses different IPs, user-agents, and other methods to mainly disguise itself as a machine service is unethical in this context.
And you are a lawyer?
>This directly contradicts your previous comment
Absolutely not. The default intention of putting an HTTP server online is not "letting stuff lying around", it is publishing stuff. And yes, the default can be overruled in various ways.
>There is already case law precedent regarding this exact type of publically accessible information not being authorized
You're grasping for straws here. In this specific case, it was completely obvious that this information was not supposed to be public. It was an embarrassing security failure that the defendant wanted to expose.
I think we agree on a lot. robots.txt should be honored and scraping in way or for a purpose that negatively impacts the website's viability or business model is unethical. But usually, such purposes are covered by copyright law anyway.
That doesn't mean that it doesn't happen.
I agree with the GP in that public content is fair game. How do you thing google works?
Copyright governs how the content is used, including distribution. The reason people who download videos are not liable is because you have to download the complete content to see the copyright. File sharers have already downloaded the content and are subject to copyright. Bots that scrape can interpret meta tags in the header of the dom, which is why scraping and violating copyright is unethical.
In terms of copyright, what matters is what you do with the scrape. If you scrape a public website for personal use, it's no different from just browsing it for personal use. If you try to republish the content for your own benefit, you'll run afoul of copyright law.
I use to think web scraping was limited to obscure proprietary companies like connotate so I'm glad to see more of these tools becoming available for everyone to utilize and hopefully create something cool.
As some has pointed out scraping is not exactly a new thing and a lot of the biggest sites out there are built on the basis of web scraping or crawling. We provide a tool and expect you use that tool while abiding the law - and if not we will of course shut your account down immediately. Breaking the law includes violating copyrights and performing DDoS attacks (Although they will be rather small attacks since even 50 concurrent agents is no big deal for most websites).
We consider ourselves good netizens. We wish nothing more than to provide a good, easily accessible and safe tool for extracting valuable information from the internet, be it for a price comparison site in a market that lacks transparency, business intelligence for your company to make informed and wiser decisions, or a PhD project that requires access to millions of data points available online in unstructured form.
Additionally if you feel we're providing services that has ill-intent - we are not providing any services (Captcha and proxy rotation) that anyone with a bit of programming skill can not easily use in their own software. The main difference is that we are actively improving and focusing not only on making a good experience for our users - but also on minimizing the impact on the sites being scraped. This involves several things like automated throttling and slow-site detection, request caching, and blocking requests to services such as google analytics - to not interfere with site owners stats.
Contrast with Feist Publications, Inc. v. Rural Telephone Service Co., where it was ruled that it was legal to copy data from a phonebook and republish, since it was non-copyrightable factual data.
There are several other ridiculous early rulings that were made while the internet was still coming of age, and I think before many judges really understood the way it worked. Recent cases have been bucking these precedents, but you can still get the book thrown at you based on those rulings.
Read about 3Taps and please understand that you will be sued, as they were, unless you fold the moment you get a C&D, which would make your site fairly useless.
Google and all other search engines are illegal in the US in most cases. They just don't get in trouble for most of their activity because people usually want to be on Google. If you end up collecting data in a way that someone doesn't like, things won't go so well for you. See Facebook Inc. v. Power Ventures, Inc.. That guy got raked over the coals; I'm sure Facebook was trying to make an example of him.
Data portability is a threat to the business model of many web incumbents, and that means they want scraping, a critical tool for ensuring that portability, to remain in a nebulous grey area; this allows them to use it for their own purposes (which they often do) and also to try to block people who are using data found on their platform in a way they don't like. This basically results in the bigger company getting their way, because only other multi-billion dollar companies really have the resources to fight against the army of $1k/hr lawyers that public companies hire to try to enforce their opinions on upstarts.
What we really need is serious internet law reform that favors a fair and open platform. Unfortunately, whenever we hear about "internet law reform", it's skewed to the interests of the megacorps who want more tools to shut down innovators that may threaten their business models, not toward creating an open and fair environment for innovation.
Consider, for instance, how ridiculous it would be if every time you opened a book one of the title pages contained a "Terms of Reading" that bound you not to use the information in the book, even the non-copyrightable information, in any way that the book's publisher didn't like, required you to only read the book using the publisher's approved reading methods (perhaps only Oakleys and Ray-bans are publisher-approved eyeglasses, only Herman-Miller publisher-approved seating, and only GE bulbs publisher-approved lighting), required you to agree that you'd never sue the publisher in court but always use private arbitrators that the publisher can easily, even implicitly, buy off, and so forth.
Consider the viability of the argument that you committed copyright infringement by looking at the pages of the book when the author didn't want you to, because the reflection of the content on your eyes constituted an illegal copy.
These things would get laughed out of court, but the digital equivalent is frequently upheld when it comes to online activity.
I think eventually things will stabilize and scraping non-copyrighted data will unambiguously not be a crime, but unfortunately, I think it may still be a few more decades until that happens. I really hope your company is able and willing to help us set the right precedents by committing the tens of millions it will take to win each piece of that stability, since you're set up so perfectly to be the target of several scraping-related lawsuits.
Recent rulings, like QVC v. Resultly and Nguyen v. Barnes and Noble Inc. have been much more positive than former ones, even if they're not altogether ideal, indicating that some magistrates are starting to think of the internet in sensible terms. The rest has to be done through the legislature. Please help make the web safe for data.
IANAL
Thank you.
EDIT: found that in your FAQ:
"Since disclosing IP’s and user agents would allow anyone to identify all traffic coming from our system – we naturally never do."
That is the opposite of being a good netizen and I hope I'll be able to sue you once I find out your services are helping to scrape my content.
2nd EDIT: Found out that you reside in Denmark and therefore in the EU, that makes it way easier then.
Humans don't read pages at dozens per minute.
Not hard to code for that, especially if your static content like images are served by alternate servers, so you can just focus on your content servers.
The problem I am dealing with lately is massive bot farms where you do not see a repeating IP for an hour. I catch them eventually but it takes a toll.
But this is why you preemptively block all of aws and servers like them.
http://www.h4labs.com/dev/ios/swift.html
This is a weekend project so I don't want to spend a lot of money on it.
You may also get sued...
If the site owner wants its data to be available for automated extraction they'd provide an API and can price it to compensate for cost of serving all those bots.
Countless sites depend on web scraping. You can scrape and be a good netizen, the two are not mutually exclusive.
If someone needs to build a search site for real estate etc why couldn't they just scrape the Google search result, filter it (white list domains), extract the actual link and present it? In that case you'l need a Google specific scraper that can be based on open source scraping libraries.
Update: Google actually will IP ban you if it thinks you're a bot trying to scrape for search results.
But they have an API:
Second, you linked to an api page that clearly says it was deprecated 5 years ago, come on. No one should ever feel bad about scraping Google, considering Google is the world's largest scraper themselves.
- Automatic IP rotation
That's just wrong. Is there a website where we can blacklist IP addresses of such violators ?
You haven't explained how is it wrong and why. None of those thing are "wrong" by itself. It is the malicious use, of any tool, that is unethical.
Is there a website where we can blacklist IP addresses of such violators ?
What exactly do you think is being violated here?
If this has automated CAPTCHAs, there will be work into making much more difficult CAPTCHas. I don't know if this is sustainable. I remember 2 different sites where I havent been able to solve their captchas
First thing, it's just a suggestion. Don't use duplicate account to reply to the comments. Do it from your original account.
I think what the user "chdir" means by wrong is that you're not honoring "robots.txt" and how do you account for ethical scraping ( eg. running 50 concurrent connection on a single website and overloading the website, technically its DoS attack)
If I were a content owner/producer and I see automated scraping from IP addresses owned by Cloudscrape that violate the ToS, I would sadly treat the entire pool of IPs as violators (even though some might be genuine users who are respecting the limits).
I'd like to know what's a legitimate use case of auto-resolving captchas and IP rotation other than circumventing limits imposed by webmaster.
P.S. Why the throwaway ?
While I agree that captchas and IP blocks can be employed by target sites, I don't agree that it should be illegal to circumvent them. I also don't agree that it's necessarily unethical (though in some cases, it may be). If you have public information posted on the public web, I don't think you have the right to mandate that it only be accessed by certain tools. You should plan and expect that it will be accessed by every tool capable of doing so.
If something is disrupting your business by "clogging the tubes" or whatever, that's another thing, and they can be held liable for that. But it doesn't matter that they clogged the tubes with one type of program or another; what matters is that the tubes were clogged by their actions, and that's the part that should be focused on in the subsequent legal proceedings. The specific tool or tools used to clog the tubes is at most a tangential curiosity. We don't want to make certain programs illegal.
Maybe we need a new amendment with "the right to bear code". We do not want to get down a rabbit hole where certain programs are legal and certain programs are not (at least not anymore than we already are with the DMCA et al). Down with code control!
Appeal to fear.
No comment.
No being able to determine a right cause doesn't prove a wrong.
Loaded question.
Often they ignore robots.txt also, they still continue scraping, sometimes they even publish things in their SERPS.
From Google's FAQ:
However, robots.txt Disallow does not guarantee that a page will not appear in results: Google may still decide, based on external information such as incoming links, that it is relevant. If you wish to explicitly block a page from being indexed, you should instead use the noindex robots meta tag or X-Robots-Tag HTTP header. In this case, you should not disallow the page in robots.txt, because the page must be crawled in order for the tag to be seen and obeyed.
Also, do they respect robots.txt? And how are they going to avoid being blocked by websites that don't want to be scraped? (I guess it would be easy to determine the IP addresses of their scrape robots).
Wouldn't be hard to file abuse@ reports to their hosting provider as well. Just pipe their scrapper IPs into WHOIS.
I then wanted to back out, of course there is no way to remove your account.
While I know what scrape/robot/etc does, I don't know how to start using this one after spent some minutes reading through on the website.
What you put online is public. Treat the web the same as a public bulletin board. Everything you post will be readable by everyone. Trying to prevent this is pointless.
What's the difference between a machine scraper and a human scraper? Humans use machines to access websites. All your users are machines. And behind every machine is a human.
If you don't want someone to access all your data in a short time-frame, then don't give it all away in a short time-frame.
If I just download a site – for local use, or similar cases – it’s not an issue.
And if I provide a scraper where the user customly has to enter the URL, it’s also technically just a very advanced webproxy, and also okay.
Think about it from the point of view of the site being scraped. Suddenly your webserver is being slammed with traffic that's way beyond what any human would cause. It's costing you bandwidth and compute resources. You have every right to say no, scraping is not allowed (in your Terms of Service). It's then up to a court to decide if those Terms of Service are legally binding. Some courts have decided that yes, they are.
Since you're clearly not a content creator, and have decided since you didn't work on it it has no value, but putting something online does not make content public domain.
Google scraps webpages as a core competency.
I might just want to train a neural network on extracting data out of websites, and, for that usecase, wget -m every website I can find.
How I use the data you present to me, as long as I don’t republish it, is my decision. If you give me a license to read it, you also give me the license to copy it onto up to 7 different media at the same time, and to show it to up to 7 friends at the same time.
My site, my robots.txt - if the scrapers do not obey, then I'll block them.
There are ways to force authorization, if you don’t use them, your fault.
If I just use TOR or VPNs or similar to get around your block, also, again, your issue. Unless you require authorization, your service is, by law, public.
And if I want to store the site offline for later reading, or read it now in a browser, is nothing that you have to decide.
No, it's not "by law" public, or could you cite a few laws for that, both US and EU. Would appreciate that.
HTTP directly says you should use authorization to make stuff non-public.
And third-party content scraping tools are NOT violating copyright law, as they do not redistribute it freely.
Otherwise services like Opera Turbo – which scrapes a website, removes cruft, compresses data, and sends it to your phone – would also break copyright law, and there is a nice exception for this case, under US law, it’s covered under fair use.
(Unless the scraping service makes your content accessible publicly – and even then, fair use applies usually, as in the case of pure archival like archive.org or archive.is)
Are you a lawyer? I am not - anyway, I can't really agree with that falls under fair use.
If I charge money to scrape someones site and then hand over the material in whatever form my customer uses - I am bound run into problems sooner or later.