This is one of PG's frighteningly ambitious problems and I believe it's an important (and really hard) problem. I think the answer is to focus on an important niche and grow from there. DDG has done a good job by focusing on privacy; they're currently at nearly 4m queries per day.
I'm starting by building the search engine I want to use while writing code, which does parallel searches of different parts of the web dynamically based on the query.
I have an initial (ugly and very limited) prototype here: http://gigglebang.com/
I'm looking for co-founders. If this is an interesting problem to you, email me: gb@dewdrops.net