My web scraping toolkit (Python):
-Beautiful Soup
-Requests Lib
-JSON Lib
-Selenium
-Urllib2
-Cookielib
Handles 99% of things I encounter. With dynamic sites, you're often better off simulating the request than controlling the browser.
Then you get to parse your way through someone's annoyingly formatted JSON.