but hey no problem it's JS, so to correct that SEO problem they will probably do a whole render of the page server-side with something like PhamtomJS and serve those "special" pages to bots
which I think is totally insane
Explanation of it: http://searchengineland.com/how-to-use-fetch-as-googlebot-li...
go there http://web.archive.org/web/20160119113309/http://cdmedia.git...
view source, that's what archive.org_bot is referencing eg. no content whatsoever
if/when this site die, when you will browse the link above, you will see a blank page
I would even say that on web.archive.org the effect is even more vicious, if you don't know any better, because you see the page archived you think it is indeed archived when in fact it's not
but don't despair, there are other way to have the exact same result for your SEO
tell me how it is different than rendering the whole page as Flash SWF file ?
GoogleBot can read/parse SWF and even understand when dynamic text/xml is loaded https://googlewebmastercentral.blogspot.fr/2008/06/improved-...
since 2008