Hacker News Reader

Comment by bovermyer | Hacker News Reader

I'm curious about what it would take to build my own "toy" search engine with its own index. Anyone ever tried this?

Yeah that's where I started out in 2021. Been at it for almost 5 years now, last three of which full time. I'm indexing about 1.1 billion documents now off a single server.

Hard part is doing it at any sort of scale and producing useful results. It's easy to build something that indexes a few million documents. Pushing into billions is a bigger challenge, as you start needing a lot of increasingly intricate bespoke solutions.

Devlog here:

https://www.marginalia.nu/tags/search-engine/

And search engine itself:

https://marginalia-search.com/

(... though it operates a bit sub-optimally now as I'm using a ton of CPU cores to migrate the index to use postings lists compression, will take about 4-5 days I think).

Curious on what (how much) hardware your running this.

Currently running off

AMD EPYC 7543 x2 for 64 cores/128 threads

512 GB RAM

~ 90 TB of PM9A3 SSDs across 12 physical devices

Storage is not very full though. I'm probably using about a third of it at this point.

I assume you able to monetize it since you work on it full time?

I'm mostly living off grants and donations at this point, but the plan down the line is to polish it up well enough to make some money off providing an API like the one Google is making it a hassle to access with this change :-)

Might find YaCy interesting. It’s meant to be a decentralised search engine where users scrape the internet and can search other users indexes in a kind of torrent like way.

I found it didn’t really work as a real search engine but it was interesting.

Good luck scraping websites without being blocked, if you're not Google.

Well you'll get blocked some places but it's not too big of a deal. If you're running an above board operation, you can surprisingly often successfully just email the admin explaining what you're doing, and ask to be unblocked.

Sounds very time consuming. Glad you're able to sustain yourself to be able to do it full time.

Reply on news.ycombinator.com