There was a number of problems, but mostly boiling down to the sheer scale of our deployment. As an engineer I would say that KT gave us surprisingly good mileage. We were able to grow the company from couple of servers to hundreds, but at some point we just outgrew KT.

The main issues that triggered serious internal discussions about KT:

(1) Tuning the data structures was hard. Each data backend has many toggles, and it's not obvious what they actually do. I ended up writing script to try out all the different toggles just to figure out what the disk and performance cost would be.

(2) If we decided to tweak the data backend toggles, it was not obvious how to upgrade the storage. There isn't a single command to take the KT store and rewrite it with different specific-optimization-setting. This meant our disks stored stuff in non-optimal format, wasting our disk and memory.

(3) We added encryption to the replication protocol, it was fine for some time, but not perfect. If I remember right the SSL handshake was blocking, which meant the master stalled for some time on each replica connect. This code was ours, sure, but the problem is: integrating non-blocking SSL accept() into KT event loop was very much non trivial.

(4) The replication protocol is great (and trivial!), if you the lifetime of the replicas is roughly in line with the lifetime of the master. This was not the case in our deployment. We might bring a new colo up, or just format disks in some region. In such case, how do you bootstrap the replica? This is an unsolved problem. Sadly, you can't just rsync the data directory while keeping KT master running: it will end up inconsistent. We had to: stop the master, rsync the disk, start the replica, start the master, make sure the replica picks up the log in right place. This is fine if you have two replicas, but not hundreds.

(5) The HA story of the "KT master" was unsatisfactory. The docs say that you should configure two masters, each replicating to each other. But I never understood how it was supposed to work in case of conflicts, clock skew, or disk failures. Think about problems in second or third layer of replicas.

Most importantly (4) and (5) lead to data inconsistencies, which is unacceptable. Furthermore as we grew we were constantly toggling our KT architecture, adding more and more replica layers. This made (4) only worse.

I personally love KT, I think it's great piece of software. Sadly, it looks abandoned, lacking community. I guess at some point for us the pains of operating it outgrew the benefits it gave.

Our replacement, Quicksilver, is brilliant. Perhaps we should blog about it more.