I think I would just start with indexing the latest Common Crawl WET files along with a server and a couple of Exos X20 20TB drives.
Wondering out loud if there's a crawl that will produce real-time updates via common news outlets..
Wondering out loud if there's a crawl that will produce real-time updates via common news outlets..
The 20tb drives would be too slow for query though. Query really needs everything in memory to be fast enough.