How much money is in DuckDuckGo, Mojeek, Gigablast, YaCy, or other minority search engines? If indexing material requires serious resources, the only viable search engines will be the huge ones.
Sites would just have to decide which search engines they want to allow to spider them, which will probably be a hardcoded list in a webserver config file somewhere, with the associated public keys.
If we're lucky, webserver package maintainers will try to keep these lists up to date, and Google may even contribute some funds for the development effort, doing just enough to stop this system from triggering an anti-trust investigation.