Hacker News Reader

There's around 50M rooms, and the dataset is back to 2014.

However, much of the room is taken up by the Synapse DB schema being particularly denormalised (prioritising performance over disk footprint) - especially caching snapshots of historical key/value state for rooms, which currently takes up ~65x more space than then actual underlying dataset. Ironically, we're looking into that currently, but not fast enough to speed up this DB rebuild.

View on HN

So that means it's possible to reduce the DB size under 1TB, your recovery time would take less than 1 hour, rather >17hours.

Thanks for that explanation. It makes a lot more sense now. Back in the day I managed a Jabber + bandersnatch corporate server and it had many years of Jabber + AIM + MSN + ICQ + Yahoo chat but the database was tiny which is why I was very surprised to see a chat server take up so much space.

Good luck on getting the schema overhead out of the way. I'm sure nowadays you are probably also using faster underlying storage SSD's behind the raid controllers. Dell/HP keep them overpriced of course but I found them to be very much worth it for databases as did the DBA's.

I hope your on-call teams get to take a week off after that incident.

Reply on news.ycombinator.com