why is it assumed the web ought to be crawleable?
Plus there is already an endless sea of walled stuff (paywall, "create an account and login" wall, "fill in your address/ZIP code and we'll give you the best price" wall, government filtering wall).
I'm not saying it's great, or it would be great, but currently having 1 (or maybe 1-and-half) search engines is also very far from ideal.