Introducing the Priceonomics Analysis Engine and API

Thanks for the feedback! Right now the Social analyzer pulls how many people shared that one specific page on FB (or liked it or comment it), rather than how many followers a Page has or Users an FB App has. We could add that per your suggestion though.

We currently use it to see how popular various articles are on the web since FB likes is a good proxy for overall traffic. FB like are public whereas page views typically aren't.

The main reason we think there needs to be a platform for crawling and structuring data is that it's a huge hassle to reinvent the wheel every time. If you have experience writing crawlers and have been able to afford the time to learn to write them well, you've basically already reinvented the wheel. But now you also have to maintain the wheel. We think it would be better if people started working together on the problem of structuring the web. Besides running infrastructure (which has it's own challenges), there is really no reason why everyone needs to build their own extractor for every website. People generally want to extract the same things, and just a handful of quality implementations should be enough to satisfy almost everyone.

One of the keys to actually having an extractor repository is providing incentive for people to build and maintain the extractors, which is actually where we're headed once we allow developers to start building their own applications on the Analysis Engine.

If you're good at extracting structured data from HTML, you should continue writing your own extractors and even consider selling access to them. On the other hand, if you can't be bothered learning the art of extracting data, why not pay someone to use their commercial grade extractor?

So, to summarize, you should use our API so that you can better leverage your time and contribute to the creation of even better structuring tools.

We really like the stuff that Kimono and Import.IO are working on. They are significantly lowering the barriers to getting started at extracting/structuring data, which is great for everyone. Of course there are limitations to what their tools can do, but that's what you'll see every time someone attempts to simplify a complicated process. We aim to be the glue that connects people with data acquisition, structuring, and analysis tools.

Crawling social media is pretty tough because social platforms tend to be very reserved about access to their data (probably due to the enormous amount of personal data they have). It's a disturbing trend for sites to require an account before they even let you past the landing page. It's basically not publicly available data anymore. That's actually not something we have much experience doing, since we haven't had any customers ask for it yet.

Thanks zeeshanm! We actually have a lot of parsers that do just that, we just haven't yet made them available on the engine. We're also looking at ways to let people create their own and make it available to other users, since we think that's the best way to get to a fully structured web. Are there any sites in particular that you think we should focus on first?