Shameless self-promotion, but important point nonetheless:
VoltDB is truly fantastic at this kind of workload. It's one of the first use cases we had traction with. Also, VoltDB is open source today.
VoltDB is truly fantastic at this kind of workload. It's one of the first use cases we had traction with. Also, VoltDB is open source today.
For example, the March release of VoltDB will support more than 6 table joins and has some improvements to aggregation and distinct code. It also has a much more usable explain plan feature.
We feel that VoltDB offers one of the richer query interfaces of systems that scale to its level, but we don't plan to sit still.
- VoltDB isn't log-structured, so you really only have to store the state. How fast you can mutate it isn't limited by RAM amounts. We see use cases with utter firehoses of data that update just tens or hundreds of gigabytes of state.
- Beyond normalization, you can probably reduce the number of redundant counters, e.g. use SQL to count which URLs start with "amazon". This would be painful in many systems, but depending on the query, can often be done at scale in VoltDB.
- The byte overhead per counter is also likely much lower in an ACID/Relational store.
Finally, VoltDB is designed to migrate data to disk based stores (such as Hadoop or an OLAP store) as memory fills up. This is a feature we're working very hard on see as a big differentiator. It adds complexity if you need to query across stores, but you get a best-of-both-worlds feature set.
Is the database really entirely in-memory, though?
It also supports continuous and transactionally-isolated snapshotting to disk.