Lazy Redis is better Redis

> "Redis is very fast as long as you use O(1) and O(log_N) commands"

Well, like you say, to me that particular practice would be pretty obvious. More helpful would be a page that shows the time and space complexity of the various commands, akin to the following page for Python:

https://wiki.python.org/moin/TimeComplexity

Edit: Yeah, like the Redis documentation, lol. I think it says a lot for Redis that I almost never need to visit the docs, other than the single page that lists all the commands (back when it was red was the last time I actually needed to look at it). Once you know the commands, quickly reading the changelogs provides me with all I need. Anyway, all that to say I had no idea Redis already provided this complexity information, don't remember them having this back when I first used learned Redis (although maybe they did). Good to know.

This line jumped out to me as well. After a recent production incident determined to have been heavily influenced by several thousand O(N) commands being called against a list several orders of magnitude larger than normal, I'm thinking about this a lot.

I'm confident many other users of Redis are using O(N) operations in production against small datasets without knowledge of how much latency will be introduced by those operations when that dataset grows. This is exactly the kind of situation that makes me immediately skeptical when I find Redis in a emergent system design.

I'm considering what initially felt like a draconian means of remediation: using rename-command to rename all O(N) commands to BIG_O_N_$COMMAND to ensure everyone using them knows the possible impact and to allow for easy detection during code review and/or Redis latency spikes.

The more I think about it this approach, the more I feel that this should be the default mode of operation for Redis in production. SREs around the world would collectively save decades of time if the every engineer writing Redis queries to knew this fact by heart:

> Redis is very fast as long as you use O(1) and O(log_N) commands. You are free to use O(N) commands but be aware that it’s not the case we optimized for, be prepared for latency spikes.

Hello nxb (but this comment also addresses jnewland post). In the documentation we have, for each command, the time complexity, and in certain commands like KEYS an explicit mention about not using those in production.

However the best tool for debugging if there is something wrong with a given deployment is to use the LATENCY command. Once the latency monitoring is enabled, by calling LATENCY DOCTOR Redis creates an automatic report about all the possible sources of latency with a given application/setup by collecting and analyzing live data.

The latency reporting is explained here: http://redis.io/topics/latency-monitor

Moreover sources of latency and a quick list of things to check to make sure to avoid 99% of the problems are also listed here:

http://redis.io/topics/latency

The first section is a TL;DR synopsis listing everything very important.

In general the "slow log" feature, the latency doctor and reporting abilities, the software watchdog, and the documentation which explicitly provides time complexity for each command, should be quite a good amount of tools and informations in order to avoid problems. Obviously like with any popular tool like Redis, there is a percentage of users that will not read the doc and just deploy it. This is unavoidable but at least when they run into issues we try to provide them with the tools needed.

Agreed. Worth to note it's always better to have at least two cores per Redis process if persistence is enabled since there is the saving process that will burn a single core from time to time.

The key is removed from the main hashtable and thus not accessible by the normal code path. It's appended to the internal list for deletion then. That's basically just moving around a pointer.

A manual rename is not needed.

Absolutely not, it's the memcached model. There are N threads (like 4 or 8 or one per core) serving all the clients in a multiplexed way like is happening now with a single client.

If we'll go the extra mile implementing also background slow operations, we'll send them in another "ops" thread, will block the client, and will resurrect the client and send the result when available. Like we do currently for blocking operations like BLPOP.

As specified in the article DEL remains a blocking DEL. It's application code that calls UNLINK if a background release of memory is needed because of low latency needs. However 50% of the blog post explains how the implementation avoids to create problems by releasing objects faster than you can allocate new ones in order to avoid the undetermined memory behavior[1]. Anyway memcached is a lovely product, I hope it will be developed more in the future so that users have a choice. But in the context of our discussion, there is no equivalent in memcached for this feature since objects are composed of a single allocation. You get the same if you just use Redis SET / GET / DEL commands.

[1] This is also handled in the LRU code. If there is nothing to free but there are objects in the free list, the server waits for the background thread to make progresses to continue again. For expire and eviction of keys there will be explicit options where the user specifies if server-deleted objects should be deleted in a lazy or blocking way, with the old plain blocking as default.

memcache operations are mostly constant time, but their behavior is not what I would call deterministic due to the slab allocator. A long running memcached will not behave the same as a fresh memcached because pages are not reclaimed from a particular slab class.

Say you have a memcached that has been running and only storing 80 byte objects. The 80 byte slab class has all of the memory pages, and the other slab classes each have one page. Now your key size increases (for instance, because it includes the uid) and your objects fit into the 100 byte slab class. You will only be able to ever store 10,000 of these objects while the rest of your memory allocation is effectively unused.

Not an obvious problem to diagnose. You can only fix it by restarting memcached (and losing everything) or (very slowly) migrating pages one at a time to the larger slab class. This anecdote is based on a true story.

So your SET operation is still constant time, but the behavior is not deterministic. redis doesn't have this type of issue. I wish that I hadn't wasted so many years struggling with memcache.

> a backend who's performance characteristics change with the dataset & operations being performed

Isn't that all of them?