Hacker News Reader

Check out Mechanize: https://github.com/sparklemotion/mechanize

I haven't used the ruby version but I am pretty happy with the python port of it. It's lighter and faster than phantom, but it won't do javascript interpretation.

I found a tutorial making it look pretty easy to do with Mechanize: http://readysteadycode.com/howto-scrape-websites-with-ruby-a...

Note, not tried this myself!

While I haven't tried any, I think if you want to handle dynamic Javascript content, you'd have to go with a JS library. Feel free to correct me if I'm wrong.

you can do it in pure ruby with one of the webkit wrappers (i.e. poltergeist[0])

[0] https://github.com/teampoltergeist/poltergeist)

I've always used Mechanize+BeautifulSoup in Python.. I think Mechanize has also a ruby lib...