Ask HN: Is 30k QPS Per Node enough? | Hacker News Reader

Ask HN: Is 30k QPS Per Node enough?

2 pointsronnykylin1 comment

Developer of Apache Doris (an analytic database like ClickHouse) here. Kinda proud of what we've achieved lately but also want more input from HN. These are the nine methods we've done to make "30,000 QPS per node" happen (ignore the promotional discourse if it bothers you): (https://doris.apache.org/blog/High_concurrency) trying to collect advice for improvement, like what you are looking for in an OLAP engine?

View on HN

Depends on what is size of said node and what type of queries.

Example of something I was struggling in the past:

  - 200milion-4bilion rows dataset with properties like country, production, x etc 
  - support arbitrary filters on any of 70 columns
  - provide result in less than 4 sec and support different aggregations

Example of problematic query:

Calculate monthly aggregated (sum) time series of X for all where country is Y and another attribute is above Z. In Elasticsearch this query takes 10-20sec and with heavy caching it somehow works. It was hard to partition data in a way that was friendly to all queries.

Reply on news.ycombinator.com