Clearly lots of people are trying but they have very strict anti-crawler measures that often trigger for me (a human) if I browse too quickly.

Well that's the huge irony. Ancestry paywalls almost everything useful, so you need to set up a monthly subscription to see the stuff.