EDIT: first commit 22 hours ago - goes to show that when you have thought about the idea and know what you're doing it doesn't take long to produce the first version. :)
lxml.html is much better in my experience. If you want to use CSS selectors, there's pyquery.
lxml.html has a terrible parser.
> If you want to use CSS selectors, there's pyquery.
CSS selectors are built into lxml through cssselect[0] which is used to convert CSS3 to XPath 1.0 selectors.
[0] If you want to use CSS selectors, there's pyquery.
I’ve found more luck with html5lib and cssselect2.
Now, I very rarely have any need to scrape and parse html data, and I was scratching my head at how it was taking these parsers so long to parse a 3.5 mib html page. I mean, it should be able to go through that and get what I want in under a second right?
So, I said screw it and wrote some regexes. 10-15 seconds was how long it was now taking to parse that html. It actually took 1.5 or so seconds to parse the html; the rest was waiting for it to download the webpage.
Ironically, implementing the regexes was actually quicker than figuring out how to use those html parsers and write the code. Of course, that's assuming you know how to craft regexes. Since it was set-up to run every 5 minutes, I wanted something that could do it without spending 1/2 the time parsing the data(amongst other tasks the processors were needed for).
YMMV
It's when you don't have guarantees about the structure of the html you are working with, that regex will come up short.
I heard KRE was going to headline at the upcoming PyCon.