100GB takes 6 hours? That's ~4.7MB/sec - is that right? That seems, sorta like slow?
Not to sound rude, but what is so difficult about managing 100GB of data? You can fit that in RAM without much work.
Not to sound rude, but what is so difficult about managing 100GB of data? You can fit that in RAM without much work.
As for the 100GB, I have 12 task servers with 6-8 processes each reading 25-50 stories each every 4 seconds. I also have 18 app servers with 6-8 processes each reading 12 stories every second. Each story is on average 4KB. That's several MB a sec.