Using jQuery and node.js to scrape html pages in 5 lines
blog.nodejitsu.com
blog.nodejitsu.com
the article you clicked on was most certainly not submitted by nodejitsu.
news.ycombinator has a long history of squashing articles and submitters that aren't funded by y-comb.
most of this is done through their "silent" banning and censoring mechanisms, that leave people not even realizing they have been silenced.
i hope you enjoy this article, and remember that HN is extremely biased and that you should keep your horizons broad."
While I would agree that HN is bias towards YC-funded projects I would not agree that it is biased against non-YC projects or news. In fact, the majority of the items on HN are non-YC. This also follows for submitters and commenters for the year or more I've been here.
On a different note. Hpricot is not representative of Ruby scraping anymore - nokogiri (http://nokogiri.org/) is where it's at. Which has a Hpricot translation layer if you need to change. Even when I decided to solidify on Python for everything else I will still go back to Ruby just for nokogiri when it comes to scraping.
It doesn't happen too often, however. The community here usually has pretty high commenting standards, and there's peer pressure that encourages good behavior.
They of course aren't notified at all, so they're quite happy to keep adding comments to the site, all of which stay at 1 point and never get replied to (or even read by anyone except those with showdead on).
Since there's no messaging system on this site, if they haven't bothered to add any contact information to their userpage then there's no way to let them know.
I occasionally see these folks posting some pretty thoughtful comments while auto-dead, too.
However, this has been a subject of discussion here a few times, and nothing has changed; I don't expect it to.
I do expect that this site will be banned soon too.
I'm not saying that that's the best solution. A mefi-like $5 cost to join might be better, but then a) PG has to implement the payment stuff, and b) some people will complain about us charging nontrolls to get accounts.
It's possible that there could be some system where if, say, 5 people with at least 200 karma will vouch for you, it's free, but then there's even more complicated code that needs to get written, and people will complain that you can only join HN if you pay or are part of the "in crowd" that know lots of HNers.
I actually think that could work really well though. If for the first month/50 karma, those people were supposed to moderate the sponsorees comments and give them feedback on what they're doing well and what they're doing badly, we might have a lot fewer complaints about all those noobs that are ruining our site.
Whether being a jerk justifies banning I can't say - but his assertion that HN is biased has little justification (particularly when you consider that the writer himself is biased.) Kindly ignore.
It kinda comes down to what sort of "thing" HN is supposed to be -- whether it has a place for people like him -- but regardless, he's certainly not exactly hurting over his inability to play in our little sandbox here.
Knowledgeable he may be, but professional he is most certainly not.
Edit: I must admit, however, that I'm interested as to why he was banned. I don't believe he's said/done anything spiteful and mean in HN itself.
He's been a major drain/drag on the Node.js community, and makes the IRC channel a toxic wasteland for a good part of the day.
He trolls other sites as well... http://www.youtube.com/watch?v=IrkDqh9ZVog
I find his hook.io project (http://hook.io/about.html) particularly interesting, and the BDD testing app he was building for Node Knockout has a lot of potential as well.
Marak comes across as someone who's seen that and intends to emulate it, without the actual development skill or talent to back it up.
He sends his regards, and if you'd like to contact him visit the #Node.js IRC channel @ Freenode
As far as I know this is impossible with any other server-side scraping technology.
If so, that would be amazingly useful for a couple of my side projects, much easier than parsing their Javascript code and extracting the info I need.
I imagine Google has something at least close to that internally, given that they've dropped hints here and there about their ability to crawl post-js-processed pages, but unless I've missed it, I don't think they've released anything (my guess is because of the arms-race issue, with shady sites trying to use js to mask certain content from googlebot).
Selenium might be OK for usability testing a single site, but it's useless for large scale multithreaded crawling and scraping and doesn't scale.
I, for one, would definitely pay good money for an easy library/API to scrape JS-heavy sites without the overhead of a full browser running macros.
For portability, performance and flexability I finally settled for writing my own HTML parser and CSS selection engine from scratch.
And that's true only if you only want to do page manipulation in Javascript. I'm perfectly happy with my page manipulation in Ruby w/ Nokogiri. Here's an example:
(code formatting on HN sucks, so it's on my blog, apologies)
http://blog.knowtheory.net/post/1074676060/xml-manipulation-...
1. It was a 48-hour coding competition, so some bugs are to be expected.
2. The hosting for the apps in this competition was turned on literally the day before: http://twitter.com/joyent/status/22204412477
I don't think that "node can't scale" will get much press, just because node is pretty performant and many people conflate performance and scalability. Node's asynchronous api would seem pretty good in a services-oriented architecture, and scalability is mostly about how the app is architected. (and in 48 hours, that probably means "quickly")
On the other hand, node.js is still a very young platform and there are a lot of unsolved issues. As of right now there are only 3 or 4 dedicated node.js hosts that I know of, and all are in private beta. People are still trying to figure out best practices for design and testing.
The thing that I took away from Node Knockout was that here are a bunch of really cool web app ideas that, given time, could be built with node. I think the apps were 'frozen' pending judging, but I know many of the top app teams were planning on improving them as soon as they had the chance.