In a discussion about this at work a few months back, someone suggested to me that part of Google's motivation might be to increase the cost of entry for search competitors. No longer can you just scrape a page and index it: not one of the 22 KB served at that URL actually contains the text of the page. To scrape one of those awful pages, you need to JavaScript interpreter and a DOM - a headless browser, really. Unless you're Google: they can just index the back-end directly

I'm not entirely convinced, but I've heard worse conspiracy theories that turned out to be true.