Jonathon Fletcher: forgotten father of the search engine (2013)
bbc.com
bbc.com
What's interesting about web crawling is that it has always been controversial.
Many website operators back then did not like crawlers. And that sort of sentiment continues to this day. The web has never fully consented to being indexed.
This has always struck me as odd. Perhaps there should have been more thought about a bifurcation of "private" web and "public" web. Of course in retrospect the value of the public one seems far greater; Google's corporate wealth points to that conclusion.
But I think Jonathon would agree that it was the obvious next step. There was already the (overly) academical WAIS etc.