Noodlejs - Node.js web scraper | Hacker News Reader

Noodlejs - Node.js web scraper

noodlejs.com

90 pointscharlieirish19 comments

I didn't submit this, but I did oversee the project.

Noodle is a Node-based web scraper that also handles JSON, XML and other file formats. It was initially built as a hack project to replace a core subset of YQL.

All responses can be served as JSONP, to allow for cross-domain scraping from a website's front-end.

Selector queries can be used to grab a subset of a document - e.g. CSS selectors for HTML documents and dot-notation for JSON documents. It lets you request multiple documents in a single HTTP request, and a few other things.

I helped to guide an intern, Aaron Acerboni, at my company, Dharmafly, when he built it last year.

I can recommend formatting the JSON. Like this:

  demoElement.innerHTML = JSON.stringify(data, null, 4);

And then use "<pre>" tag in the demoElement.

That is the first thing I did using the web inspector. One could also use a js-beautifier I guess.

A good idea. I've just added that to the docs site.

For those looking for the source, you can manually get it:

http://npm.im/noodlejs is the NPM package

https://registry.npmjs.org/noodlejs/-/noodlejs-0.2.0.tgz is the package source

    $ curl -kO https://registry.npmjs.org/noodlejs/-/noodlejs-0.2.0.tgz

The code is pure-JS (not a C++ addin)

I get a 404 when I try to access the project's GitHub page or download it.

same here. (probably, they've not made the repository public yet)

I too got the same error. Seems like there is some issue.

Ditto

Whoops. Now fixed. Not sure how that happened.

README https://npmjs.org/package/noodlejs

Npm package npm install noodle

Npm tarball curl -kO https://registry.npmjs.org/noodlejs/-/noodlejs-0.2.0.tgz

noodle should be noodlejs

Unless I'm reading wrong, this scraper is a slightly prettier API to what most do: a simple curl, structuring the data with selectors.

Like most it ignores page requiring state or anything dynamically generated on the client.

How does this compare with Cheerio? ( see discussion on HN from yesterday - https://news.ycombinator.com/item?id=6273905 )

I can't answer your question without reading further, but I note that Cheerio is a dependency of Noodle.

It uses Cheerio for scraping, but it also lets you use selector queries to grab a subset of documents, whether the document is HTML, JSON or another format. It also lets you request multiple documents in one HTTP request, and a few other things.

how to get the list of urls in this example on http://noodlejs.com/?

the selector: 'h3.r a' allows only to get the list of names

Simply replace `extract: text` with `extract: href` for the anchor's href attribute. You can do this with any kind of attribute.

This is very cool! Thanks for sharing.

Reply on news.ycombinator.com