Show HN: ScreenSlicer – Automatic, zero-config web scraping | Hacker News Reader

Show HN: ScreenSlicer – Automatic, zero-config web scraping

github.com

26 pointslogn17 comments

Using neural nets and tuned heuristics, ScreenSlicer is able to intelligently find a search box, enter a query, extract the results, and page forward in the results.

This sounds like it might be usable for making a universal program to go through the idiotic, annoying, hated-by-everyone web-based sign on screens required by many free Wi-Fi hot spots.

As in, something that doesn't have a rigid template database of what UI it can deal with so the users have to beg "please support such and such hot spot in the next release".

Deleted comment

There's nothing proprietary in the entire stack and there's no SaaS being offered here. The API is separated from the core logic to allow for clustering. You can put both API and core servers on the same machine.

Specifically, the neural nets are used in finding the parent node of the result set (or one of the parents). Heuristics (basically just generic strategies) are used for finding search boxes. There's also a similar function for finding authentication forms. Most of the functions in this project are static so it should be easy to use isolated components as needed.

The first thing I tried it on was http://hcpdirectory.cigna.com/web/public/providers, which is a typical instance of a site I might like to scrape, but still one of the simpler ones. It was not able to extract meaningful information.

That works in my test locally. The cloud servers are rather overloaded at the moment and also sometimes Tor is just flaky. Here's the request/response: http://pastebin.com/raw.php?i=KncMgbfB

edit: note that there's a couple different search forms on that page. If you run this on your own machines you can use the Form Query API which lets you specify a form ID and then set specific values for each form field (e.g., for a provider search with City/State and Name). And you can use any non-Tor proxies or a direct connection.

What an offensive, restrictive, and anti-free software license! Too bad - this might have otherwise been a useful piece of software. The current license renders it toxic.

The GNU licenses are arguably the definition of free software. More permissive licensing just gives you the freedom to make software non-free. That's also offered for this project, but as a paid privilege.

It is not unfree or anti-free at all. You can use and copy it and modify it and share it, unlike proprietary software.

To those curious, the license is the GNU Affero General Public License version 3.

It is a very open-source unfriendly modification of the AGPLv3 license.

The modifications (available to paying licensees) are designed to let this project be useful to those running scraping SaaS's. It lets them hook this into their proprietary infrastructures without GPL'ing their whole stack. Revisions/additions to the project itself need to be licensed as AGPL but the rest can remain proprietary.

The AGPLv3 license stands on its own without the modifications (they're optional). It's essentially a dual licensing option.

There are a limited number of ways to do open source and make a living at it. And given that I am in business independently with no investors, I have not many other options. The one path I've ruled out, at least for now, is BSD-style licensing, as that just allows SaaS operators to leverage my work, deny users freedom, and also not pay me for my time to help their commercial projects.

Ahh, the Peanut Butter Hula Hoops crazy licence: http://www.billthelizard.com/2012/05/which-open-source-licen...

The more I think about it the more I like this dual licencing setup which seems a realistic way to ensure those working on OS get paid.

It lets everyone experimenting enjoy the code, whether they are a destitute student or an aspiring startup.

Meanwhile, if you are someone who wants to actually make money from this AND want to hide your own code, you've got to pay the piper.

Otherwise you end up in BSD land (or is it MIT land) where everyone takes from your project and give nothing back.

Is there part of this that I could use for "main text content" detection on generic web pages? (i.e. not the search results listings)

I recommend Boilerpipe, which happens to be embedded in ScreenSlicer too: https://code.google.com/p/boilerpipe/

You're supposed to build an audience first before you lock them in.. not do it right at the beginning.

Looks like it's getting crushed at the moment.

Reply on news.ycombinator.com