Cache Reheating - Not to be Ignored
blog.mongodb.org
blog.mongodb.org
This is not correct, even if the operating system handles the cache it needs to reheat the pages that are used in order to cache the memory mapped file.
So usually for instance you have a big MongoDB database but the application accesses the database in a very biased way, there is an obvious working set. Part of the working set ends being cached on memory pages by the operating system virtual memory system, so many times when you do a database read operation you read reading from memory.
If the server is rebooted and the MongoDB process restarted the file is not cached in memory, so the first access to every different "page" of the file will actually seek and read from the disk.
For example, at Google my caching strategies could be exported in production with only a few minor changes to the code. If we experienced data center downtime I could actually offload a hot cache to the other backing stores in round-robin configuration. The problem isn't when one database server gets taken down, it's when all the servers in the datacenter get taken down at once without any warning.
And I agree it is an advantage that at least in this scenario no reboot is needed, for instance if you plan to upgrnde MongoDB version. Actually when you can plan the downtime it is usually simpler to handle in the general case of many cache hosts running at the same time when you can do the upgrade incrementally, but it is nice that a single big server can be upgraded without the problem of the cache reheating.
Apologizes again for not reading the article with more care.
Even if ALL your cache servers go down, you'll at least have the reheat load distributed among a cluster of cache servers in a super-efficient direct memory dump rather than hitting your DB servers over the network.
I'll build a big db and then run ad hoc queries interactively and be like... this is pretty cool. Then I'll reboot, start the query, go get coffee come back and see that it's still running.
On a similar vein, I like to partially randomize my cache invalidation period if I don't have the luxury of using some other cache invalidation scheme(http://the-robot-lives.com/index.php/2010/03/fractional-keys/). These prevents cyclic cache invalidation which can lead to interesting issues such as having your server fail every 24 hours after you initial cache restart when a huge some of 24 hour cached items invalidate and start hitting the database at the same time, or whenever the hourly, daily and short duration cache items all invalidate at the same time.The numbers presented only make sense in the event that your cache hit rate is uniformly distributed.
In the event that you have a normal distribution or pareto distribution (more likely for a website) the majority of your cache hits come from very few entries meaning your cache can probably be reheated to 80% with 20% of the data, or 200GB instead of 1 TB.
A much better idea that actually solves the problem instead of discussing implementation artifacts (memory mapped files) is to reuse your tracking data (eg. this page gets X number of hits) to pre-heat the caches before the server goes live.
All you need to do is build a histogram of your cache hits over the last X minutes and flush it to disk every now and then.