Redis Cluster Re-Implemented in Rust: Scaling Redis Easily in Kubernetes | Hacker News Reader

Redis Cluster Re-Implemented in Rust: Scaling Redis Easily in Kubernetes

github.com

106 pointsdoyoubi21 comments

Like Tendis? https://github.com/Tencent/Tendis

Tendis is a high-performance distributed storage system fully compatible with the Redis protocol.

Tendis focuses more on persistence and it uses decentralized architecture just as Redis Cluster.

I don't know a whole lot about Redis, so I'm struggling to understand what this is. Am I correct in saying that this is just the clustering mechanism? So I'd need to run Redis instances, and use this to cluster them? If so, would this work with KeyDB (would that even make sense)?

Yes, that's accurate. This project runs on top of Redis as Redis has already done a good job as a memory storage engine. And https://github.com/doyoubi/undermoon-operator will help run it with Redis instances together, so you won't need to set them up manually. And yes, it can work with KeyDB in theory.

Is Redis still single-threaded? I thought that would become a bit of a problem by now.

Redis 6 has already supported multiple threads for IO. The memory storage engine is still inside a single thread but I think it's fine.

Do you mean that a single CPU can saturate the throughput of a machine's RAM? That's a bit surprising, given that current crop of Intel CPUs does non uniform memory access. Or, is that bandwidth so much above any I/O that it doesn't matter?

Nop. It's that a better solution would still be using multiple Redis instances on each physical server. Trying to support multi-threads in the storage engine requires complex synchronization implementation. Sharing nothing between each small instance achieves higher performance and is a lot simpler. Also, FULL SYNC between masters and replicas can be uncontrollable. And smaller memory size for each instance alleviates the performance impact from FULL SYNC.

not to mention, multiple instances follows the indistry direction in horizontal scalability. Common advice when building for k8s style cloud environments is to target single core, n instances, to maximize scaling flexibility and atomicity.

There is a parallel but less fancy reimplementation of Redis called Pedis:

https://github.com/fastio/1store

It's built using Seastar, which is sort of an ugh but cool unholy marriage of node.js and C++14 with an event loop per thread, and message passing between threads. It predates C++20 coroutines I guess, so you pass callbacks around like in Node using C++ lambdas. Maybe they have updated that by now.

They have also done a version of memcached using DPDK. I don't see a Pedis one, but check out their memcached one (scroll down): http://seastar.io/

Cool! Thanks for sharing.

Wow that's a product name horribly prone to typo/auto-correct awkwardness.

At first I was like, “heh, parallel redis so pedis”…then I said it out loud and I was like….oh my

> It's built using Seastar, which is sort of an ugh but cool unholy marriage of node.js and C++14 with an event loop per thread, and message passing between threads. It predates C++20 coroutines I guess, so you pass callbacks around like in Node using C++ lambdas. Maybe they have updated that by now.

That seems like a lot of work just to badly recreate Erlang. (Which, yes, doesn’t let you define your own native data structures in-and-of-itself — but there’s always NIFs, e.g. https://blog.discord.com/using-rust-to-scale-elixir-for-11-m....)

The master/replica setup is interesting because we did something similar at Shopify, but I don't believe this is conventional for redis cluster. Can you expand on that a bit? eg: Looks like master/replica are together on a host, wouldn't you want them spread out?

You are including proxies for clients and server? How do they compare to envoy? eg: Are you doing anything clever to handle failovers transparently for clients?

Is the metadata storage just using redis?

I have to admit the diagram is a bit misleading.

The core idea here is called chunk: https://github.com/doyoubi/undermoon/blob/master/docs/chunk....

Each proxy always runs in front of 2 Redis instances. Each chunk consists of 2 proxies and thus 4 Redis instances on 2 different physical servers. And each Redis has its master/replica on another half part under another proxy inside the chunk. So master and replica are always not in the same physical server. Chunk is the smallest building block of a cluster so the Redis instances number is always a multiple of 4.

We designed it in this way because we came up with an algorithm to balance the overall workload based on this chunk structure: https://github.com/doyoubi/undermoon/blob/master/docs/chunk_...

For the second question, the proxy here is just a way to add additional features on the top of Redis without changing its source code. I would say it's a server-side proxy while envoy focuses on the client-side features.

And yes, it can handle failover transparently for clients.

In Kubernetes, the metadata is stored inside a ConfigMap, which is based on etcd.

How does it work ? Is it sharding different keys to different Redis instances ?

Yes! It uses exactly the same slot model in the official Redis Cluster. Each key will first calculate its slot number by a hashing function. Then try to find out the instance with this corresponding slot. So this solution is compatible with the existing Redis Cluster clients from the community.

Makes perfect sense :)

This allow scaling without impacting the consistency guarantees of redis !

I might have an use for it since we will have to expand our Redis instances to clusters at work.

Feel free to let me know if you get any questions :)

Reply on news.ycombinator.com