Hacker News Reader

Comment by davidhollander | Hacker News Reader

>I can't wait for the persistence issues to be worked out.

Why use an in memory database for persistence instead of for cacheing? Considering that RAM is the scarcest and thus most valuable resource on nearly every server, in memory DBs like Redis are better used for storing already rendered data ready for output rather than every bit of raw data that always needs to be duplicated or logged on disk at the end of the day anyway. Persistence==disk!=memory.

>Redis Diskstore is something I'm eagerly waiting for

If your data can be represented using hashmaps (unordered) and b+trees (ordered), check out TokyoCabinet for on-disk persistance. It has the fastest on-disk hashmaps and b+trees (of non-fixed size) and has been around for a while.

View on HN

The point is Redis is beginning to move away from being just an in-memory database. See my other comment for a more detailed answer as to the use case.

Persistent caches are an important commodity for many websites. For some services trying to handle standard traffic patterns with an empty cache is suicide and can cause cascading failures across the board. Without a strong persistence solution Redis can't be used here. antirez mentioned he considered it a strange move for Reddit to use to Cassandra and not Redis as a persistent cache[1] and I think the issues with Redis persistence may well have caused that.

Additionally I still think there's reasonable ground for a database that's primarily in-memory but drops least used data off to disk. The vast majority of the web follows a Zipfian/long tail distribution so although your "working dataset" can be far larger than RAM your actual "active working dataset" can fit in there. Why trade away the advantages of an in-memory data structure driven database when almost all your queries can be satisfied in this manner?

[1] http://www.reddit.com/r/programming/comments/bcqhi/reddits_n...

>for a database that's primarily in-memory but drops least used data off to disk

>so although your "working dataset" can be far larger than RAM your actual "active working dataset" can fit in there

To me that's the exact same pattern as cacheing, merely stated in reverse. If you want full persistence, everything's going to have to hit the disk anyway no matter how you phrase it and how and when you store it. In regards to your user logs problem stated below, that sounds easy to parallelize. I would hash+modulo the username to a data server number, on the selected data server traverse a B+ tree of dates\entries in order, and buffer or stream until 20 entries matching the usernames have been retrieved in response. High performance and fully persistent.

Waiting for a new memory+disk database pattern that does everything more intelligently and faster than every other disk+memory pattern seems like procrastination of parallelization, which is the unavoidable long term answer to these problems.

Reply on news.ycombinator.com