Where is the cost coming from? Wouldn't a crawler mostly just accessing cached static assets served by CDN?
And what do you mean by your search infrastructure? Are you talking about elasticsearch or some equivalent?
And what do you mean by your search infrastructure? Are you talking about elasticsearch or some equivalent?
I didn’t work on search, but yeah, something like Elasticsearch. Googlebot was a majority of our search traffic at times.