Tokyocabinet + LZ4 = Speedy Gonzales
blog.creapptives.com
Using a simple compression technique can result in a huge difference.
blog.creapptives.com
The cache values were in the 2-8K range and GZIP gave me about %90 compression. I also "compressed" the cache keys (which could be in 1-2K range) by using their MD5-sums instead of the actual key. The tiny chance of a collision was hugely outweighed by the memory savings.
But what about very short keys and values? Things on the order of 10 bytes?
I imagine that the compression header would not allow for meaningful compression.
What would be an appropriate compression library for this use case of compressing short texts? Particularly if we know that we are compressing English text? I have searched for such a library, but cannot find one.