tsergiu

One of the creators of debuggex here.

For javascript, debuggex conforms to the regular expression implementation in google chrome (which is mostly the same as all other browsers).

Specifically, a \n in your regex where group n doesn't exist is actually interpreted as the octal \00n.

If I recall correctly, \1(a) should match just "a" because all groups start out as being empty.

Parsing regular expressions is indeed complex, and we've got a massive suite of tests to make sure our engine works correctly (with respect to the chrome implementation, since that's what most people are using js regexes for).

Here at ParseHub, we're attacking the problem from another angle. Our tool lets users turn semistructured html (even the complicated, interactive kind) into structured data. The user does this by visually describing relationships in and across pages.

The key idea is to make it easy for another party to add the semantics on top of your data. This solves some fundamental issues that you and Cory Doctorow mentioned:

1) The economics equation for tagging now works out. The user that's doing the tagging has an immediate need (and payoff) for doing that tagging.

A corollary of this is that the parts of the web that are most valuable (in the sense that users need them the most) tend to get tagged first.

The following are responses to Cory's essay:

2.1) The person that's doing the tagging is also an end user, so there's an incentive to do the tagging honestly. That doesn't stop the underlying website from lying. But that's an issue with the web in general, and is mitigated by things like SEO penalties, reviews, etc.

2.2) Again, the tagger is the person who benefits from the tagging, so as long as the data is valuable enough, it will be tagged despite laze.

2.3) We haven't overcome human stupidity. Presumably since the person tagging the data needs it, it will be at a "good enough" level to be usable.

2.4) This one doesn't apply; the tagger is a different person.

2.5) 2.6) and 2.7) These are tougher, and we haven't started working on them yet. You have the same problems when trying to consolidate data from multiple sources. One possibility is to have several alternatives and allow searching to choose between them. That's how Bloomberg solves some of these problems, though it does result in fragmentation.

I'd love to talk to you about this some more. You can email me at serge@parsehub.com

Full disclosure: I'm one of the founders of http://www.parsehub.com

One of the founders of ParseHub here.

Not quite. Sikuli tries to figure out where things are by doing a visual match. This works very well for things like automating applications or sites where page elements are fixed (e.g. finding an option in a menu or using a search engine). But it works terribly when trying to overlay semantic structure on dynamically-generated data. For example, it has no way of knowing that a list of movies is split up on multiple pages, with each movie having multiple genres, a cast, and multiple reviews, each of which has a rating and an author.

There's also the additional drawback that it is hard to parallelize things in Sikuli (you would need heavyweight vms, and there are no obvious "breaks" in the flow). So doing something at scale is not feasible.

With ParseHub, one of the goals is to make it easy to express relationships (and we think we've done a really good job). We also automatically figure out how to split a job up across an entire fleet of servers.

Hope that offers some insight. Email me at serge@parsehub.com if you have any other questions.

Sorry for the delayed response. See my answer to a similar question in this thread: https://news.ycombinator.com/item?id=8356999

If you have any more questions or just want to chat, feel free to shoot me an email directly at serge@parsehub.com

ParseHub itself runs on a highly optimized browser engine. The idea is to give a visual interface so that you don't have to worry about low-level details of how to control the browser. This also makes it easier to reason about the logic of what's happening since humans have stronger spatial reasoning than symbolic reasoning.

Yes, it does! Check out the interactivity tutorial in the extension.

Great ideas to consider for the future!. We may decide to release an offline version of ParseHub eventually.

We run a headless browser to execute your extraction.

We've invested very heavily in building out a solid infrastructure for extracting data. We want to make sure that the product Just Works for our users, and that includes rotating IP addresses (you don't have to fiddle with your own, we have access to a pool of thousands).

Robots.txt is a tricky balancing act. It was first conceived in 1994, and was designed for crawlers that tried to suck up all the pages on the web. ParseHub, on the other hand, is very specifically targeted by a human. A human tells ParseHub exactly which pages and which pieces of data to extract. From that point of view, ParseHub is more like a "bulk web browser" than a robot.

Here are some examples that make this line blurry. If I tell ParseHub to log into a site, visit a single page, and extract one piece of information on it, does that violate robots.txt? If yes, then your browser has been violating robots.txt for years. The screenshots of your most visited websites are updated by periodically polling those sites (and ignoring robots.txt). My browser is currently showing a picture of my gmail inbox, which is blocked by robots.txt https://mail.google.com/robots.txt

More importantly, your computer and browser already do a lot of robot-like stuff to turn your mouse click into a request that's sent to the server. You don't have to write out the full request yourself. Is that then considered a robot? If not, then why is it considered a robot when ParseHub does the same (again, assuming a single request) thing?

Furthermore, some sites don't specify rate limits in robots.txt, but still actively block IP addresses when they cross some threshold.

It is far from a perfect standard, so it makes a lot of practical sense to have the ability to rotate IPs, even if it's not appropriate to use that ability all the time.

Our goal here is to be able to distinguish between the good type and bad type of scraping and give webmasters full transparency. Obviously this is a hard problem. If you have any feedback on any of this we'd love to hear it.

ps. we've tested our infrastructure on many Alexa top 100 sites and can say with moderate confidence that it will Just Work.

pps. if you're a webmaster, having ParseHub extract data from your site is probably far preferable to the alternative. People usually hack together their own scripts if their tools can't do the job. ParseHub does very aggressive caching of content and tries to figure out the traffic patterns of the host so that we can throttle based on the traffic the host is receiving. Hacked together scripts rarely go through the trouble of doing that.

That doesn't work because you need the javascript as well.

Sorry about that. We haven't tested with Tor.

I hope disabling Tor is suitable for your use case.

Do you have some time to show me over skype? We've done extensive user testing, but may have missed a few bugs. Email me at serge@parsehub.com, skype id is t.sergiu

We chose Airbnb because the complexity of their site highlights the flexibility of our technology. That being said, we also assumed they wouldn't mind because of their origins :)

Yes, ParseHub works with login forms (they are no different than regular forms). Check out the interactivity tutorial. If you still don't get it after that, I'd be happy to show you 1-on-1 over skype.

Please note that the password will be accessible by ParseHub, since it needs to enter it on the web page.

Currently, we support local deployments only in our custom enterprise plan. That may change in the future.

Crawling already works :)

One of the things that has been heavily marketed by other web scrapers is "crawling" as a separate feature.

With ParseHub, all the tools easily combine, so you don't need that distinction. You can use the navigate tool to jump to another page (see our interactive navigation tutorial in the extension for the details).

And you can combine multiple navigations to go as deep in the website structure as you like. For example, say you have a forum that links to subforums that link to posts that link to users. You can easily model the structure of such a site by using a few navigation nodes (one from forum to its subforums, another from subforum to posts, etc.). The result would be a big json (or csv) dump of all the data on the forum, in the proper hierarchy.

We've really tried to make our tools as general as possible. A side effect of the navigate tool is that you can use it to get "pagination" for free as well (another feature that's been heavily marketed).

Didn't find an email in your profile. I'd love to discuss with you further. Can you email me at serge@parsehub.com?

Sure, I'll give you one example.

We want to show a sample immediately as a user changes what they extract. On a static website, this is fairly easy. You simply run what the user created on the currently visible page.

However, when you involve interactivity, you can no longer do that. The major problem is idempotent operations. Imagine a click that changed the dom of a page. And now imagine running the sample on that same page. Re-running the sample may no longer work, because the click could have changed the page in such a way that the extraction no longer works (e.g. it deletes an element from the page).

To solve this issue, we actually reset a "hidden tab" to the starting state of the page you're on. This happens every time you re-run a sample. Unfortunately, it's not possible with Chrome to create such hidden tabs. We also mess with the cache to make sure that this tab can be reset really quickly, something that we couldn't find an API for with chrome.

Hope that answers your question.

Thanks, we'll look into it.

A fix is on the way!

Chrome extensions run in a severely restricted environment. While this is arguably good for security, it prevents us from building some of the powerful tools we can build in Firefox. We do plan to eventually release as a standalone app with no browser dependency.

Sorry, we'll make the wording more clear.

At the moment, ParseHub does not respect robots.txt. We do expect to add this + features for webmasters in the future, but have not had the developer cycles to do this yet.

We have a fully-documented API that you can use to integrate directly with your app. https://www.parsehub.com/docs/ref/api

We think Kimono is a great tool, but it is very limited in capability.

We specifically focus on handling highly dynamic or interactive websites. Our toolset gives you more flexibility over how you can extract data. For example, you can extract all the nested comment data from a reddit post, or you can extract data from maps without having to fumble around in the web inspector.

One of the founders here. Our goal with ParseHub is to enable data extraction not only from static websites, but from highly dynamic or complicated ones as well.

We've done this by separating the concepts of selecting and doing something to a selection. Specifically, we've created tools like click, input, hover, etc. that you can combine with any selection, and with each other. This keeps a lot of the power that you have with programming.

This also applies to the data structures that are created. So it's easy to express nested lists or even recursive lists, because of the ability to combine tools easily.

If you have any questions I'd love to answer them.

Thanks! Pushing a fix now.

Edit: Fixed

Related: http://www.debuggex.com/?flavor=python also does visualization in addition to matching (full disclosure: built by me)

Click "Export to StackOverflow", then you can save the image.

Thanks for the feedback! I'll take this into account for future development plans.

Can you give me a use case for this? It is not very hard to do, given that random matches are already being generated, but I'm skeptical of its usefulness.

Thanks! I'm very flattered that you created an account to leave a comment :)