I'll just say. Google offers and maintains reCapcha, while at the same time scrapes the internet using bots.
Wedging yourself in the middle is an awesome business plan.
Wedging yourself in the middle is an awesome business plan.
[1]: OK, maybe in the extremely broad, indirect sense that reCaptcha helps Google solve image classification problems, which make Google products like Maps/Books/Images better, which increases traffic, which increases ad revenue
Google is involved in making it harder to scrape the web for the average person. They are involved in most web standards. If you go in to most robots.txt files, they have root forbidden for user-agent: *. But you can be sure that Google is whitelisted.
Google has a vested interest in making it harder for the average person to scrap the web. This is because they position themselves as a repository of information. This is a clear case of wedging yourself in the middle.