> Only a select few crawlers are allowed access to the entire web, and Google is given extra special privileges on top of that.
Hmm, so set up a VPN on the Google Cloud so you have a Google IP address, use a Google User-Agent, and go!
Hmm, so set up a VPN on the Google Cloud so you have a Google IP address, use a Google User-Agent, and go!
describes the procedure for checking "is this source Google it". You couldn't fake it just by running on gcp