Hosting costs are so minimal today that I don't think crawling is a natural monopoly. How much would it really cost a site to be crawled by 100 search engines?
Bots are typically tuned to work with generic sites over crawling efficiently.
And what do you mean by your search infrastructure? Are you talking about elasticsearch or some equivalent?
I didn’t work on search, but yeah, something like Elasticsearch. Googlebot was a majority of our search traffic at times.