Ask HN: Replacement backend for RRD
I am currently re-engineering an stats tracking, analysis, and alerting application which is based upon rrdtool. There are currently roughly 20k data points being monitored every minute and pushed to rrdcached. I'm seeing a substantial performance impact as the quantity of data points grow, even though I am running it on a fairly big box, as the initial implementation of our app targeted roughly 1000 data points.
Ultimately I'd like to not stress about scaling the application to 100k data points. I've thought of using Redis, PostgreSQL and Hadoop/HBase, all which are tackling different domain problems. One of my concerns is if I remove the lossy trending of rrd, the size of the data footprint is substantial at 20k data points (20k * 1440 * data lifetime * per-row-overhead).
Do you have any suggestions on a data backend, preferably something lossy like rrd?