Tidis - Redis protocol on top of TiKV

Sometimes, for various reasons, on-prem is the only choice. Politics, bandwidth limitations, expediency, egress charges, heavy-handed IT security, etc. Similar for EC2 clones.

Your view point of just using something already hosted by GCP/AWS is probably the default today for good reason. I would break down the reason to not do that to three reasons: cost, performance, and customization. Lets ignore cost (which is complex) and focus on the TiDB and AWS example.

TiDB was build to operate at a scale for OLTP (scale writes horizontally just by adding more nodes) that doesn't exist on Amazon's MySQL. You would have to you DynamoDB and downgrade from SQL transactions to eventual consistency.

In terms of customization, TiDB is offering something AWS does not: the ability to store ever more of your data in one distributed data store (TiKV) and to leverage that in different ways (query it with TiSpark, and perhaps more projects like this in the future).

I’ve used all of the amazon datastores and none of them write as quick as a data store designed like this or maybe something like Hbase or Cassandra depending on what you are doing. Something like this is even better. I think amazon does Hbase as a service btw.Those use cases are rare so Aurora is usually the right answer. Alternatively A hundred s3 buckets can handle a large amount of writes as well for large or medium sized writes but it’s annoying to manage.

Well, my business is working for a cloud services provider, so I can’t exactly rely on cloud services.

Complex services are hard to manage, even at scale with billions of investment. We've had plenty of instances where managed services had issues but VMs (being relatively simple tech) were just fine and we would've been better off running it ourselves. Also there's the cost ratio where high performance becomes exponential more expensive for many services.

Perhaps the biggest reason though is that software is easier than ever to operate now with the rise of containers and Kubernetes. In fact, you can use Kubernetes as the single managed service to get that taken care of by the cloud, and then do the rest yourself for a great balance of cost, performance, flexibility and reliability.

Pretty soon you'll have to write the whole thing in rust/c++ depending on how much efficiency you really need.

At least move many things in coprocessors in tikv layer.

Not directly related to tidis, but I did read about another Redis-on-top-of-Tikv use case by Ele.me, which is apparently this big food delivery platform like Doordash that's super popular in China. Here's the post: https://zhuanlan.zhihu.com/p/35226447 (too bad it's in Chinese right now. Hopefully someone translates it into English soon...)

Many in-memory databases don't have native data types for sets, sorted sets, hash tables, bitmaps, HyperLogLogs. Postgres and Redis have different use cases.

Our use case for redis is strictly in-memory cache only, basically what it was originally designed for. It’s used solely with the caveat that it cannot be trusted as a source of truth, that its data may disappear at any time, that the data is local to the node it is running under, etc.

For us, redis takes the place of IPC and “persistence” by not sharing lifetimes with its consumers, ie while redis May go down and our microservices restart all the time, the two have separate lifetimes and redis being around between the two bridges the temporal gap.

Using redis to store structured data is ineffective once you need something even remotely structured, requiring too many commands to perform what should be an atomic operation.

Redis excels at sub-millisecond operations - thats what you get when you use an in-memory data store. If you are conflating Postgres and Redis then it's doubt you really actually needed either.

Strong persistence or consistency isn't something that is very popular in Redis (or at least I haven't seen it done). In most cases you are right - if you are willing to trade latency for persistence or consistency there are other tools available.

I wouldn't read that much into tidis - it seems like a hobby project for now. Another interesting redis project is pedis (https://github.com/fastio/pedis). It takes advantage of multiple cores by, at a high level, sharding the keyspace per core.

You're specifically talking about a relational database which are the most flexible data systems available with decades of development. You can model pretty much every other datastore on top of a relational database, including Redis.

But Redis is designed for a different use-case It's far simpler to install and operate, and much faster to run by being completely in-memory. It has optional persistence when you need it, easy replication, and a simple single-threaded architecture with primitive access to data structures. So yes, you won't get the flexibility of a relational database, but you get performance and simplicity and there are a great many uses for that.

If you can't think of a reason to use it, then that's a good sign that you just don't have a need for it.