Show HN: Ruby gem to scrape a web page

Good work, but you might not have heard about Mechanize: https://github.com/sparklemotion/mechanize

I was surprised there's no way to query the page beyond the small list of element accessors you provide (body, url, scheme, host, port, title, description, links, images, meta).

When you're putting together a tool like this, it's nice to give the user some way to "escape" your framework and get to lower level underlying data.

Why not offer something like

  page.selector('h2 p') #returns Nokogiri elements
  
  page.h1 #calls method_missing, returns Nokogiri elements
  
  page.p #ditto

  page.noko #returns underlying Nokogiri doc

Also, you forgot to include the body accessor in the "Accessing inpsected data" portion of the doc.

Similar gem that scrapes OGP and oEmbed tags as well as HTML tags. Also configured using Faraday and allows for serialization/deserialization of underlying data: https://github.com/socialcast/link_preview

Always nice to have alternative. Mechanize is certainly the big player in this space, but I like the use of faraday here.

Thanks for sharing.

What does this use underneath? I wouldn't use it unless I know whether it uses libcurl or something else.