Instead of having many bots do inefficient crawling, web sites should publish their own index. Intermediate parties can combine indexes of the sites. Sites that do not provide indexes get less visitors.
That's the best way to fill your search engine with spam. There needs to be a third-party that verifies that the site-provided index is inline with the actual content. At which point said third-party can be a crawler.
This is the idea behind sitemaps which have existed forever.
Sitemaps are lists of urls on a site. They are not a text index.