In Prometheus we solved this by having Histograms where we exported cumulative buckets of latencies, which can be aggregated and then the quantiles calculated from them.

http://prometheus.io/docs/practices/histograms/#quantiles