Speaking of which, there are 'open' indexes of the web that anyone can download and use to build their own search product. But I can't remember the name of them. Can anyone help? Googling it results in dismal failure.

I think there is still huge room for innovation in search. Google has taken a different track, and they're focusing on Machine Intelligence. (in my opinion) search is just a nice side effect, but I really don't think it's their focus any longer.