KeyDB CEO Interview: Getting into YC with a Fork of Redis | Hacker News Reader

KeyDB CEO Interview: Getting into YC with a Fork of Redis

console.dev

140 pointsdmytton73 comments

The real problem is the Redis Cluster protocol, which pushes all the smarts onto the client. This means each client implementation behaves differently and fails differently.

I would encourage Redis users to use Envoy Proxy as the Redis client (ie use vanilla client, and use Envoy as the cluster client). You get all the HA and usefulness of Redis Cluster, but way less of the headache. Also, strongly encourage people to check out Elasticache, which is really good.

100% this.

And to your point about client libraries: for any shared behavior that needs to be local to the app (eg. session handling, permissions, etc.), it's more appropriate to spin up a sidecar that you can talk to over sockets than to try to build client libraries in each and every language. Your client libraries will differ in behavior and it is an incredible pain keeping them all up to date, patching every app, etc.

Envoy for S2S and traffic mesh + sidecars for shared behavior is better than building client smarts.

But the Envoy proxy only supports a subset of the commands. For example, it doesn’t support LPOPRPUSH, precisely because the keys could be stored in multiple nodes, and they don’t support that part of the clustering protocols.

I forked Redis to create Thredis almost 9 years ago. Some notes on how it was implemented:

https://github.com/grisha/thredis/blob/master/README-THREDIS

Then I added SQLite to it, more details here:

http://thredis.org

This was all done mostly for fun, though I did use it in a an actual project for a while.

Really Cool project. Just a quick thing, can you make the html responsive with the viewport meta tag?

We (me and some folks at my old consultancy) wrote an Erlang version of Redis (https://github.com/cbd/edis) for some of the same reasons - multithreading changes some of the scaling semantics in interesting ways. It was mostly for fun but ended up in some real projects as a simple REDIS protocol implementation front-end where the backend could be replaced with whatever the implementor wants.

Does anyone have any experience with these other Redis clones? I need to write a benchmark on these someday (the outline for the blog post is already written), but have restricted my yak shaving recently:

- https://github.com/Tencent/Tendis

- https://github.com/Netflix/dynomite

On a separate note, was FLASH (as in "flash memory") supposed to be an acronym? I've never seen people treat it as an acronym before.

Yes. KeyDB is legit, it works well, it's free speed in a lot of cases, the maintainer is a nice guy and is very responsive. We used it in production and it had the same robustness you'd expect from Redis.

I recommend keydb in almost every case redis is used.

Any experience with KeyDB in production? Is the Flash support in KeyDB still only in "Pro" version? Cannot find anything in the docs.

Yea FLASH is only in the Enterprise version. At a rough level the open source is the best cache we can make while pro/enterprise is the best full featured database. It’s not fully there yet but the end goal is you can get caching and persistence in one place.

What persistence options does KeyDB has that Redis doesn't?

The FLASH feature in Enterprise can send all writes to disk before replying +OK, however notably we don’t fsync (AOF is still recommended if you need that). In these deployments it’s not necessary to use RDB or AOF except for replication.

And of course we have all the existing options already in Redis.

Draftbit uses KeyDB in production. It's fast

Someone on my team looked at it, and for our workload (lots of MGETs per query) it wasn't that much better than redis. We may still go with it though, our GCP Memorystore instance is pretty much at 100% CPU constantly.

We have still yet to evaluate Redis6.

Long running commands are still a problem. We have infrastructure in the Enterprise version to alleviate this but so far we’ve enabled it only for KEYS and SCAN. MGET does come up frequently so it’s on our radar.

It's multithreaded but what about Redis running on a single core in a cluster? Like running 8 Redis on a single 8 cores CPU. I don't really understand the reason to run Redis on multiple core since you can run multiple Redis on a single CPU with clustering which will have better perforamce than running KeYDB with the same number of cores.

Take the scenario where you have an 9 node redis cluster. With KeyDB you could shrink that down to 3 nodes with 3 threads while getting the same throughput. This reduces the maintenance burden while also reducing latency. There's overhead to connecting to a different cluster server each time you access a seperate shard.

Even better you might be able to avoid having a cluster at all. For many that’s the biggest win with KeyDB.

> There's overhead to connecting to a different cluster server each time you access a seperate shard.

While I agree with your point, this concern is easily addressed with connection pools.

This what redis’s creator antirez suggests: “ the way I want to scale Redis is by improving the support for multiple Redis instances to be executed in the same host, especially via Redis Cluster” http://antirez.com/news/126

It’s basically back to the multi-threading vs multiprocessing debate that also exists in Python. With multiprocessing, you have more overhead but the client itself is simpler.

It’s kind of a weird in between where you need more scalability then a single redis but no so much that you want to go to a cluster.

Are you concerned about AWS starting a competitor to keydb cloud/have you considered modifying your license to prevent that from happening? I'd imagine that'd be important in ensuring the long term sustainability of keydb development

They have Elasticache which is always a concern. In terms of open source projects my read is they really don’t want to run their own and are much more comfortable operating open source as a service. They “forked” Elastic Search 2 years ago and basically did nothing with it until it was re-licensed a few months ago. Now that they are investing more into it I’m interested to see how they handle it long term.

This seems somewhat orthogonal to the question asked. They have Kinesis which competes with Kafka, SQS with competes with a variety of message queues, redshift, dynamo db, etc and it’s my understanding that Athena was originally a fork of Presto. So, they could definitely move more heavily into the caching space if they see there’s demand.

Elasticache is already that competitor. They are scary to be sure but they have some weaknesses. The main one is branding: Elasticache is a white label Redis not it’s own thing, to the point where their own docs use the two interchangeably.

> Now that they are investing more into it I’m interested to see how they handle it long term.

Here is an analysis of the last 30 days of activity in OpenSearch

https://public-001.gitsense.com/insights/github/repos?r=gith...

If you switch to the impact view, you can see it's pretty much one guy doing all the work right now. The impact view also shows 1 frequent, 1 occasional and 13 seldom contributors, so I'm guessing the number of people working on OpenSearch is quite small.

Note, it is also quite possible that a lot of the work is being done behind the scene, so looking at the OpenSearch repo may not tell you the whole story. And if you search for OpenSearch in amazon's job board, you find they are hiring so I guess we'll have to wait.

Note: Do not install GitSense as the docker image has an out of date license that I'll need to update when I get the time.

I can't find in documents but does multi-threading effect consistency somehow? Is there a chance that I wouldn't read what I just wrote? I'm talking about single node, not about replication, cluster etc.

If it provides same consistency, is threading like :

sock_read();

lock(datastructures);

set x=3;

unlock(datastructures);

sock_write();

Yea that was roughly KeyDB’s original design. There’s some nuance in the locking like ensuring it’s both fair and fast so P99 latency doesn’t get out of control.

In the Enterprise codebase we can take snapshots which lets us do reads without the lock but it’s a bit of work to enable for commands so it only applies to KEYS and SCAN at the moment.

can you share more details about it? I thought locks are slow, but despite locks the is latency halved here.

KeyDB uses a custom lock I wrote called the “fastlock”. At its core it’s a ticket lock with some tweaks to better tune it to KeyDB. When uncontested it’s a single atomic increment. When contested ownership changes extremely fast. If we spin too long the thread will sleep although we wait much longer than any other lock you’ll find.

When I first tried making KeyDB I used posix locks. Those were way too slow.

I am too newb to understand half of those terms :)

Do you have any recommendations for a reading material to understand the underlying concepts?

The wikipedia page on ticket locks has a good overview: https://en.wikipedia.org/wiki/Ticket_lock

Generally speaking you don't want to be dealing with this stuff unless you really have to. For 99.999% of people the locking primitives that come with your programming language are good enough.

I went through the wiki, certainly it was interesting. I believe the implementation is at fastlock.cpp [0], I will go through it. You said earlier:

> If we spin too long the thread will sleep although we wait much longer than any other lock you’ll find.

Did you do any tweaks to mitigate this?

> Generally speaking you don't want to be dealing with this stuff unless you really have to.

Ofcourse. I am just way too curious and excited to learn about these!

[0] - https://github.com/EQ-Alpha/KeyDB/blob/v6.0.16/src/fastlock....

"My thought process was simply that there is a big need here and Redis had for some reason decided not to serve it. If they won’t then we will."

I feel like this and the general tone of the article are needlessly antagonistic toward Redis. KeyDB is building their entire business off of it after all.

There may very well be a need for multi-threaded Redis, but Redis as it stands today is an amazing project and there's something to keeping it simple along the lines of the project philosophy.

I’ve learned to appreciate Salvatore’s stance on simplicity the longer we’ve gone on with KeyDB. But when I first made KeyDB two years ago I was really perplexed at the decisions he made with respect to threading.

It’s not my intention to be antagonistic. I’ve had a lot of projects over the years that went nowhere and a part of me is sad that the one with the most traction is a fork.

I don't read that as antagonistic. I think you're just being too sensitive, or reading a tone into the test that isn't actually there.

It just doesn't sound very friendly to the open source community.

"If they won’t then we will." sounds harsh imho.

You should read about the concept of "forking": https://en.wikipedia.org/wiki/Fork_(software_development)#Fo...

It's fundamental to the health and growth of the open source community.

I am aware. Community management is different from what is legally possible. Were a project owner start to act in a self-interested or malicious manner, then I think proudly and aggressively forking a project is a great idea. That's not Redis. Like I said, it may be a good idea to have a multi-threaded Redis, but Redis users tend to love Redis. I would probably lean into that goodwill instead of against it.

I don't see forking as being aggressive. It's the natural thing to do if you want to take the project in a different direction to its stewards. Often lessons are learned from that process and the learnings integrated back into the main project.

Forking is not violence you perpetuate against people you dislike.

It's a freedom; a person can disagree and go their own way instead of being hounded by zealots demanding compliance in the guise of friendliness.

I'm simply offering the advice that the tone seems unfriendly and is likely to be taken as such by Redis users (aka potential customers).

> a person can disagree and go their own way instead of being hounded by zealots demanding compliance in the guise of friendliness

This is exactly the kind of aggressive and nebulously political tone that would not help a project gain adoption. Why be hostile?

I was going to ask about where you see hostility. But judging by your comment history, it looks like you're just trolling and starting flame wars :\

Jeez i'd lose my fucking jaw laughing if antirez loses sleep on that.

I'm curious about the name. Putting "DB" in the name sort of suggests it might support persistence, more data than fits in memory, write-through, etc. Is that the case? Or is the "DB" some nod that clustering means that "in-memory" doesn't have to be ephemeral?

Or in short, where is KeyDB headed, longer term?

We have our FLASH feature which gives us a baseline persistence story, you don't need RDBs or AOFs anymore. However there's a lot of work left to do in this area.

The long term goal of KeyDB is to let you balance your dataset across memory and disk in one database. In the future I think caches will just be a feature of a more full featured database and that's where we're heading with KeyDB.

I don't think more data than fits in memory is a requirement. An important innovation in databases has been the realization that there are important performance benefits from assuming all data can fit in memory even if you still provide persistence.

Even persistence isn't really required. A pure analytics view of another database is still a database by itself, but it doesn't need to actually persist anything. It seems like querying is more important to the concept of a database rather than actual storage.

I wasn't suggesting they had to, or should happen. Just trying to understand the future plans since the "DB" term is there.

There's a post about another Redis-clone on the front page today called SSDB. Seems to be a common occurrence:

https://news.ycombinator.com/item?id=19213261

There are a few, some predate us like Threadis. Most only offer a subset of the Redis feature set where we really wanted to be a drop in replacement.

Here it is the repo https://github.com/ideawu/ssdb

this is slightly different as it is disk based. but damn, how can it be faster than RAM

Probably off-topic but I'm curious about the HPE and Huawei logos on KeyDB's website [0]. What is their involvement in their project?

0: https://keydb.dev/

They’re users. We were pretty bad at asking for logos and just started down that road. Honestly if I had one regret it’s not doing more user outreach.

Redis is one of those things where you see a car with an after-market door on it, but the door is actually a house door, cut with a sawzall into the shape of a car door.

Would this mean that in a single core or low end environment, it would be better to use Redis. I'm assume cutting out multi-core complexities would be beneficial.

KeyDB is a few % slower single threaded right now, but we have some cool announcements coming in that regard.

Some people use us for Active Rep or some of our other custom commands rather than just the Multithreading.

How does KeyDB compare to memKeyDB?

https://github.com/memKeyDB/memKeyDB

MemKeyDB is fork of Redis using both: DRAM and Persistent Memory.

KeyDB's repo https://github.com/EQ-Alpha/KeyDB

This is totally going to be a Hacker News Bingo type of question.

But has anyone tried to do a clean room implementation of Redis using Rust, but speaks the same wire protocol? You would get the zero-cost multi-threading, memory safety, etc, and it would be a drop in replacement.

> You would get the zero-cost multi-threading,

You kinda have to look at how things really work underneath before you can apply buzzwords to a database.

mis-worded. I meant safe multi-threading using rust’s abstractions that make this a lot easier and guaranteed. Is that not the case?

It's been a long time since I've looked at KeyDB, but IIRC KeyDB is just Redis plus a spinlock. It's actually still very performant. There are other "toy" reimplementations of Redis in Rust that take the same approach and aren't even as performant as single threaded Redis.

The next approach you could take is using something like Glommio and take a thread-per-core design to Redis. I think that approach has a lot of potential, but the design becomes more complex (you now need something like distributed transactions for "cross-core" Redis transactions and mutli-gets)

RonDB (NDB Cluster) takes a different approach to threading and claims it's faster than scylladb-style sharding http://mikaelronstrom.blogspot.com/2021/03/designing-thread-...

"ScyllaDB also handles complex query processing in the same engine. RonDB does all complex query processing in MySQL in a separate program." — i.e., as soon as things might get a bit hairy, RonDB punts it off-system. While a clever way to keep performance numbers high, you'd have to also then analyze the MySQL latencies and throughputs to get an overall view of complete system performance.

It's a clever dodge, but it's not a magical way to eliminate the need for those pesky CPU cycle times.

Look a little closer Peter, I doubt you arrived that fast at that conclusion being right.

It's not a dodge, the MySQL process is in the same server and can even use shared memory.

He claims in total to be better.

> zero-cost multi-threading

I think you mean zero cost abstractions. Which aren't usually zero cost, but just zero additional cost over doing it yourself.

There's no such thing as zero cost multi threading. Just tradeoffs. Rust actually doesn't help with performance here (it gets in the way often) but it definitely does help with correctness - which is truly hard with multi threaded programs.

I mis-worded. I meant safe multi-threading using rust’s abstractions that make this a lot easier and guaranteed, and zero cost meaning no overhead. Is that not the case? maybe not the case with threading.

Tokio async runtime for Rust has a tutorial in its user guide https://tokio.rs/tokio/tutorial on writing a mini-redis (https://github.com/tokio-rs/mini-redis).

I've done a C clean room version and I will say that the networking part is as important as the multi-threading the data structures: https://github.com/raitechnology/raids/.

If you go to the landing page of the above, scroll down to the bottom, there is a TCP bypass solution graphed, using Solarflare Open Onload and it is capable of running several times as fast as the Linux Kernel TCP. I didn't test Redis with Open Onload, but I'm pretty sure you'll get a similar results since TCP is a major performance bottleneck in Redis as well.

Does clean room in this case mean you didn't look at the Redis source? Is there some licensing condition where this is required?

No, I don't believe that's what it means in this case. I took the usage to mean that since Rust has memory model which is different enough from C to require a redesign.

I looked at many implementations of Redis and read many KV papers. My redesign reason was similar to a "clean room Rust" reason, I desired a memory model that used shared memory that was independent of the protocol (Redis RESP in this case), allowing multiple processes with different types of protocol services to attach to it.

"Rewrite", then.

Yeah. "Refactor" may be more accurate.

Reply on news.ycombinator.com