Its efficacy is also strongly dependent on understanding that it's a keyword search engine with no semantic understanding.
Good. I love keyword search.
"Semantic understanding" can be so biased and ... just shady sometimes.
If you lean too much into embeddings and so on, it's easy to get errors that don't make sense to a human being. It's extremely frustrating when you experience "I typed X, why am I getting results about Y?!"
That said, I think there's a sweet spot with some magic, where it genuinely just makes search better. But it's like perfume, if it's immediately obvious that it's there, it's probably a fair bit too much.
Would you be able to share some of your personal highlights regarding this?
I've partially kept up-to-date with the DIY, non-corporate search space (YaCY and friends). I'd love to understand a bit more behind the engineering decisions made when creating a search engine; it seems like a very hard problem to solve.
P.S. Marginalia is a very impressive piece of work, overall -- I've heard nothing but positive remarks from users on here. I've been meaning to try it for a while, but time constraints have... well, constrained, thus far.
> This is an independent DIY search engine that focuses on non-commercial content, and attempts to show you sites you perhaps weren't aware of in favor of the sort of sites you probably already knew existed.
I have a suggestion for the “About” section at the top of Marginalia’s landing page. I think it would read better like this:
> This is an independent DIY search engine that focuses on non-commercial content, and attempts to show you sites you perhaps weren't aware of [instead] of the sort of sites you probably already knew existed.
Showing one thing “in favor of” another seems contradictory in this case.
I tried to find marginalia on DDG, not on the first page. Google has it after some garbage. If I go to marginalia.nu I get a SSL error. search.marginalia.nu works
If i search on marginalia for duckduckgo there first link is somewhat relevant but is about the app, all the other links are related to DDG but of curious relevance.
If I search for ublacklist mentioned above, I do not see anything directly relevant.
I do think that search has gotten much worse but my ability to know the magic words like “ublock origin” instead of “Adblock” and “yt-dlp” instead of “download YouTube” and phrase my search has gotten better.
We’ve all been doing prompt engineering against the Internet-wide LLM that is the spam houses.
As much as I enjoy the notion of somehow being a 10,000X developer, it's probably mostly that modern search is a filtering problem, and MS does filtering fairly well.