While BeautifulSoup is great, lxml + xpath really is the way to go. XPath is a W3C standard and works cross language and even in the browser.

If you need a an quick way to scrape javascript generated content, you can just open your browser console and use `document.evaluate` with an XPath query.