I do think your idea is great and something with a lot of value for many use cases.
The only issue is that it won't surface certain things like restaurant contact info and Wikipedia that people often need to be a top result.
I do think your idea is great and something with a lot of value for many use cases.
The only issue is that it won't surface certain things like restaurant contact info and Wikipedia that people often need to be a top result.
This is a good idea. Basically harkening back to the old internet directory days, combined with the powerful indexing tools we have now, and then use all the great language models and topic modeling tooling to make the querying great.
Said another way: moderate and protect the index, instead of trying to clean up the results of queries.
You'd need to monitor the whitelisted sites in various ways to make sure they don't bait and switch and turn to spam later, or get hacked and go to shit.
You might even be able to monetize on the publisher side by allowing sites to pay to be indexed faster/more regularly, which could potentially (but not totally) help with spam control too. Every publisher wants traffic.
You might be able to jump start the whitelist by getting folks using a plugin so you can know what domains they spend time on in the first place and index those when signal strength warrants it. Gotta be careful about that being gamed too though. Also I think this is actually the core principal of Brave's search engine come to think of it.
I actually think a browser plugin is a core component, especially if people could opt in to contributing their pages to the index (with a method to make sure they're not logged in to anything).
The plugin would also allow people to flag websites as spam/scams/low-value, regardless of whether they're in the index or not.
What powerful indexing tools exist today? I have little context on the search domain but it's very interesting.
Who would be the curators? If you open up curation to volunteers, it could be gamed by bad actors. However if you have too small a team, the results will be limited and biased. For example, favoring the English-speaking or tech-sphere web while ignoring large sections of the web with which the curators are unfamiliar.
Perhaps machine learning could help with scale - start with a human curated dataset, then train a model on it. However that could end up getting gamed too.
Let people downvote/upvote search results as an extra strong signal on top of monitoring what they choose to click on in the result list.
Also let people block domains from searches and provide some discoverability of commonly blocked domains.
That's a key question. You don't necessarily need that many of them, though, if you're whitelisting at the domain level. A few dozen people could work giving a rating to many thousands of domains every year. An ideal number might be in the thousands.
I'd also say they probably shouldn't be in industry, because they'd have an incentive to game the index.
> Perhaps machine learning could help with scale - start with a human curated dataset, then train a model on it.
While this is true, I don't think search engines other than Google and Bing need to worry that much about being gamed. It's just not worth the effort.
The best way to compete with them IMO is to choose a slice of queries that they perform very poorly at (like non-SEO spam results) and build something 10x better. But that brings up the challenge of getting users to remember to come back to your search engine.
It's tricky for sure, with no clear answer. But for the first time in a long time, I sense that we're approaching a tipping point for Google's dominance over the web.
Some of these things can be outsourced to other services. An easy Google fallback is also a common solution for new search engines.
> But that brings up the challenge of getting users to remember to come back to your search engine.
This wouldn't really be something I'd build as a business. I'd create a nonprofit, fund it with subscriptions or donations, and cover the losses myself. I really just want it to exist for myself, and if other people used it, that would be icing on the cake.
You might be interested in watching the series "Halt and Catch Fire".
Another fan!
I also don't need fiction to know what a human-curated search engine would be like, because I was using the web in the 90s when we had things like Dmoz.
edit: softened the tone oh so slightly
I didn't insult you or your suggestion, though. I said I wasn't interested in the show and that it doesn't inform my understanding of the concept.
Why did that hurt your feelings enough to make insulting accusations about my personality and character?
That just reads as a rude way to respond to me. "Why would I be?" Because you just described a plot line of a show. It might be fun to see how someone used that plot to make a show that someone reading HN might actually enjoy.
Both sentences combined also reads as "I am perfectly content in my ways and will not listen to anything anyone might kindly suggest because I know all". Maybe because it's election season, but this unwilling to listen and stonewall is quite tiresome. So maybe just rubbed me the wrong way because of everything else and might not been taken the same way at a different time?