I think one or both were acquired and immediately shut down, but I’m not 100% sure about that.
We are doing well and are independently owned.
I think there are 3 things that contribute to this:
1. It is very easy to make a prototype that looks "magical" but very hard to build something that works in real applications. There are an enormous amount of quirks that a browser allows, and each site you encounter will use a different set of those quirks. Sites also tend to be unreliable, so whatever you build has to be very resistant to errors.
2. There is a technological wall that every company in this space reaches where it is not yet possible to mass-specialize for different websites. So even if you're able to build a tool that works very well on any individual website, the technology is not there yet to be able to generalize the instructions across websites in the same category. So if a customer wants to scrape 1000 websites, they still have to build custom instructions for each website (5-10x reduction in labor vs scripting) when what they really want/is economically viable for them is to build a single set of instructions that will work for all similar websites (10000x reduction in labor vs scripting). This is something that we're working on for the next version of parsehub, but is still a couple years away from launch.
3. Many of the YC startups you hear about have raised funding from investors and have short term pressures to exit.
The combination of the three makes it very tempting to give up and sell.
Just curious, in your experimentation, have you found it necessary to train a new model for each "category"? Or have you found a way to generalize it?
1. it's possible to make it "easy to switch" by having common building blocks and only changing the "selector" across sites - lots of companies in the space do this
2. it's impossible to do "just DOM" or "just vision/text" if you want to be able to generalize "get the price of the items"
- DOM doesn't represent spacial positioning very well (see: fixed/absolute positioning, IDs and dom changing without the visuals changing, ...) so you'd need the equivalent of an entire browser rendering engine in your "model" anyways!
- vision/text is messed up by random marketing popups (see: medium, amazon, walmart, ...), it's significantly more computationally expensive to do, and can't currently get >95% accuracy (which makes it useless, scraping needs very close to 100% accuracy in most use cases)
The problem is that when you pre-train a model, you can only solve for the lowest common denominator of what every customer might want.
In ecommerce, for example, you might pre-train to get price, product name, reviews, and a few other things that are general to all ecommerce. But you won't pre-train it to get the mAh rating of batteries, because that's not common to the vast majority of customers (even within ecommerce). It turns out that most customers need at least a few of these long-tail properties that are different than what almost every other customer wants, even if most of the properties they need are common.
And so the challenge is to dynamically train a model that generalizes to all "battery sites" based on the (very limited) input from a customer making a few clicks on a single "battery site".
Can't this be crowdsourced in some way? Having each individual entity reinvent the same wheel feels like the main problem to me. What if there was a marketplace? The ability to buy / trade / sell? Maybe subscription based in some way?
If I wanted to scrape 100 sites, it might be worth $1 per year per site. Those who put in the time make money. Those who don't have the time would pay.
This isn't a technology issue per se. It's scaling a solution to the final gap the technology can't cover. A different kind of mechanical turk?
But by definition, customers interested in long-tail attributes (i.e. virtually all of them) don't have others to source those from.
It would also lower the barrier to entry and thus increase the size of the market. Imagine if the first X sites I tired all needed more work. I'd likely quit. But if that didn't happen, I'd more likely continue.
Crowdsourcing isn't The Answer. But it's certainly a better step in the right direction.
Disclaimer: I'm a co-founder of Apify :)
I wonder what other kinds of products and services would be good for that model. In other words, would tend to be acquired for good money in order to stop them.
Potentially apocryphal example: I've heard of a certain FPGA company that bought a startup which produced FPGA compilation tools that could target multiple vendors' devices, in order to stop multi-vendor tools from existing because it made switching vendors too easy.
1. Narrow target
Your market is people who need scraped data to input into some kind of app/program/code, but don't have the resources/skills/time to use scrapy or whatever.
2. Sensitive to configuration
This is also the problem with visual code and ML apps, but you even a small issue with the source you are scraping from -- say, captcha, or login, or some weird format or css you did not anticipate -- makes it almost useless, whereas if you were coding up a solution you can (usually, not always) deal with it more easily.
Those are the reasons they shut down.
The reasons why they launch:
1. Many developers have this need
Many developers have built scrapers internally, and then used them so a lot of people have worked on this problem.
What follows from this is that they can productize it, see that other people have the need, imagine the market etc.
https://media.giphy.com/media/huDTE2qLxErrkLodUN/giphy.mp4
But also, yes, execution definitely:
https://free.cloudbrowser.xyz :D ;P ;) xx
And still