I don't work with a lot of data, but typically my decisions base on basic factors and purpose:
PostgreSQL - SQL, structured data, cannot scale horizontally
MongoDB - NoSQL, unstructured data
Redis - key-value, distributed cache
I get it that you can replace storage engine and you can theoretically get more performance, but in practice compatibility and standardization is more important, because a lot of products (including third-party) will already use PostgreSQL/MongoDB/Redis, so it's no-brainer to use it as well for your solution.
However for me to pick RocksDB or some other, new, shining database/storage engine, there would have to be more compelling reasons.