Hacker News Reader

They're inserting 5 billion events per second (for some reason), and AWS throttles everything (the following link mentions a limit of 3,500/second):

https://aws.amazon.com/about-aws/whats-new/2018/07/amazon-s3...

Also, if they're planning to do alerting, S3 would not help with that.

PUTs are 1 cent per 2000:

https://aws.amazon.com/s3/pricing/

View on HN

5 billion events per second? That seems to be too much for monitoring data even at Twitter's scale. The article refers that they store the raw data and use rollup tables for aggregate queries, I wonder how much the ELT process costs in total.

> 5 billion events per second?

When you work for a large company, you have 2 problems:

1) Every has-been manager bikesheds over metrics retention

2) you can't say no. See #1.

Indeed, the article mentions 5b metrics per minute

You write to S3 in chunks for long term storage and keep hot data live. You don’t write every metric directly. Along with compression It shouldn’t be that 5 billion/s.

Even with chunks it's prohibitively expensive at this scale, and you then need some intermediate system batching the inbound data either to disk (then why use S3 instead of a database) or in memory (more expensive, more risk of data loss).

Reply on news.ycombinator.com