As for requirements, they're very low. The machines running the crawler+parser+indexer, which achieve those speeds, are dual Xeon 5639s with 72GB ram. When the indexer isn't running, the crawler and parser use 6 cores, but only "need" 4 of the 24 total on the box. The indexer generally uses very little CPU. All of those cores are actually needed only for queries. So, the limitation on throughput, at this point, is the 1gbps connection to the box. I say 'box' singular, as the 400,000,000/day is per box.
As for maintaining the codebase, it's about 10k lines, plus another 10k of data that's referenced throughout. As I've written everything in the crawler and parser by hand. There are no magic boxes or libraries that it goes into, so debugging is working around only the logic I've constructed, as opposed to someone else's logic.
That said, I've designed it to use as few system resources as possible, unless it would run faster using a uint32_t vs a bit field, for example. If someone else were to begin working with my code, I'm unsure of what that would be like for them. I'd like to believe it's generally straightforward, but I can't say until that happens.
That said, I've also designed it to be easily updated as the way that HTML is written continues to evolve. I know that's a bit hand-wavey, but it's as much as I'm comfortable describing how it works behind the scenes, as no one else I know of (other than common search engines) parses HTML like ontolo does.
If you have any other questions, just let me know. I'll be monitoring this post today.
- Ben