McDipper: A key-value cache for Flash storage

facebook.com

30 points·jonchang··5 comments

Twitter open-sourced an implementation of this concept recently: https://github.com/twitter/fatcache https://news.ycombinator.com/item?id=5202868

I hope Facebook ends up open sourcing McDipper in the future.

I find myself not working memcached (and variants) into new designs these days, mainly because with Cassandra on the backend, there's no point -- better to spend the money on the Cassandra machines and let it manage the caching, in-memory indexing, and log-structured writes.

Memcached in that scenario just adds network overhead, devops complexity, and increases the number of things that can go wrong.

For the times when Cassandra is overkill, I still end up using Redis -- with the Lua stuff, it's ridiculously flexible these days.

Still, McDipper (and Twitter's fatcache) both look cool, and I'm glad people are thinking about SSDs as something other than faster "disks".

I've been chewing the idea of adding Cassandra to my stack for quite some time as Cassandra seems like inpenetratable moster :-)

Any pointers how to start? Also, do you have any posts on how you added Cassandra to your back-end?

I can't work out if the bit labelled "hash buckets" is in RAM, while data volume is SSD or if it is all SSD. I am guessing this is the case so it is quite similar to eg Twitter's fatcache.

It's actually all SSD by default, but you can pin the buckets into memory if you'd like. We generally do this in the memcached-replacement use case and specifically for CDN. However, we do have another use case not emphasized in the article that has a mix largely consisting of smaller values which we store in the bucket cache as direct records on SSD.

In general, if it fits in memory, it's faster to keep it there. If you don't have space for whatever reason, you leave it on the SSD.

Reply on news.ycombinator.com