I don't think the index is the challenge. Anyone with enough machines and gigabits can index a big enough portion of "the web". The real difficult part is ranking and understanding web pages themselves.