One random thought... Google goes to SomeWebsite.com. The site has only enough HTML to load a big ol' JavaScript app, which Google slowly crawls. Well, that JS app makes a bunch of AJAX calls. There's no reason I can think of that would prevent Google from remembering which AJAX calls were made, and then just crawling the URLs for those calls on subsequent visits. Why load SomeWebsite.com's JavaScript.com every time you want to index the site, when you can just remember that the JS calls SomeWebsite.com/some-endpoint.json? Sucking the JSON out of an endpoint might even be faster that indexing regular HTML. Haven't written a lot of crawlers, so I'm mostly guessing here.