Hacker News Reader

Great post!

> To keep the ‘small working set’ even smaller, we cycle these JobDBs roughly every 30 minutes. The manager cycles JobDBs when their target of filled percentage data is about to exceed available RAM.

I'm confused - why does JobDB's memory trickle up over time? Isn't it a database? Are you using MySQL's memory storage engine or something?

View on HN

There are no issues with memory utilization of the MySQL databases other than once the working data set don’t fit in memory anymore it has to be fetched from disk on cache misses, which slows down fetches. Disk utilization of the Job databases increases over time because we don’t do any deletes to avoid extra write operations (a delete is basically a write). Once a database disk utilization reaches a defined threshold it is swapped for a spare, then drained, destroyed, and reinitialised to be reused later on.

I hope this gives some clarity, let me know if you have further questions.

Reply on news.ycombinator.com