Ryan Mitchell did an excellent talk at DEFCON23 about defeating bot checks and other common barriers that web scrapers face. Excellent watch for anyone interested in scraping: https://youtu.be/PADKIdSPOsc

Shameless plug: her O’Reilly book and associated github work “Web Scraping with Python” is an excellent read.