If you're in the .NET world, HtmlAgilityPack does a great job of producing proper XML from broken HTML. CSS selectors don't always suffice, particularly with sites that don't use CSS :) Sometimes you have sites where the best you can do is e.g. get the text of the 2nd h2 header following the span with text 'X'. With some utility functions I can just write:
result.ContentX = doc.Element("span").WithText("X:").FollowingText("h2", 2);
which translates to:
//span[text()='X:']/following-sibling::h2[2]
From which my code then selects the HtmlAgilityPack InnerText (i.e. less formatting, etc.). (Practically speaking, my code also does some case-insensitive translation in there, which is an area where XPath is a bit annoying, plus string trimming, checking and propagating nulls, etc.)
In my experience the greater challenge with scraping lots of data is dealing with stuff like:
- cache disabled in response headers but you've scraped 10K pages and just discovered a page with e.g. a deformed href in an anchor (e.g. "<a href+'....'>"); after giving up trying to understand how the hell they managed that, it's not long before you're writing a crawl repository so you can selectively ignore the caching rules your proxy cache happily abides by so you can quickly restart your debug session for the next weird thing you discover (unfortunately the nature of the site has forced you to do in-memory preprocessing of 50K pages before you can do the real processing for the rest of the site because they have done some OTHER weird stuff)
- sites that treat EVERYTHING as dynamic content even though you could easily cache it... now you get to do the job of the webmaster because you have many data sources you're feeding from and don't want to hammer servers
- sites with bad links but no 404 responses (just redirects); easy to detect, but still a nuisance
- proper request throttling (i.e. throttling on the basis of requests serviced not merely requested)
- dynamically adjusting the above throttling, because sites can be weird :)
- efficiently issuing millions of requests/week to a bunch of sites and scraping data from the responses in custom formats for each site