When you said you were faking a proper agent with `requests`, do you mean you were setting the headers to look like a browser, as in here?: https://stackoverflow.com/questions/27652543/how-to-use-pyth...

That was going to be my suggestion for how to get around the anti-robot responses.