Citus has a lot of quirks and one could write a book about it. But for shortness here is just couple of the top things you should know.
- Citus is not PG. It runs on PG, it looks like PG but it doesn't behave like PG. By using Citus you lose majority of advance strategies, especially ones that manipulate data (UPDATE from SELECT, DELETE, etc.). In very short, you have to forget about everything you know about PG.
This means that you have to rethink your table schemas, distribution strategies, etc. This is common in every distributed database (No/SQL).
- Citus deployed in a cloud is a bit sensitive. As the master node keeps track of each worker in almost realtime fashion, any breaks in communication results in corrupted shards. If you don't use just one user (postgres) but have different rights for different users, fixing the shards can be a pain (if you don't pay for the enterprise version, which provides you with helpful scripts).
- Adding new worker is easy, but rebalancing them to spread the shards evenly is not (again different for enterprise version).
- PG's planner can be a bit stubborn while using citus. It sometimes behaves differently from the native PG. The result is that you have to rethink you strategies and rewrite queries up until you don't get the desired result.
- Writing functions and automatic triggers can be challenging. The usefulness of them is determined by how well you designed you original tables. Once the tables are colocated and sharded on the same column you get impressive results. Writing the functions can be challenging though as you have to deploy the function to each table. They offer a script to help you with it https://github.com/citusdata/citus/blob/a2f6b29a6d4537b626f8...
- You do want to consider how many shards you need. More automatically means that Citus gets slower, but you want to be able to grow so chose more than you have workers.
There is much more, but those are the things that you should consider. As for us, we learned to live with the quirks and now we're able to run aggregation queries through billions of rows in under 600ms.
I would suggest to consider Citus only if you're considering something like Cassandra or Hbase.