Hacker News Reader

Comment by jghn | Hacker News Reader

I've found the largest culprit here is that people don't understand how many orders of magnitude are between them and FAANG scale. It's akin to how people don't understand the difference between a millionaire, a billionaire, and Elon Musk.

So they'll see some numbers that are on the upper end or beyond what they're used to seeing, and to them that's big scale. How do you solve that? Well, you reach for the tools other people have used when they have "big scale". All the literature says you can't just increase your instance size when you have big scale, so why would you try that?

And yes, that's wrong. But it drives a lot of the mentality.

View on HN

"If you have to ask if you have big data, you don't have big data. When you have big data, you know you have big data."

Seriously, you can launch a server that has almost a terabyte of RAM! Is your business doing enough to make that database even blink? "My sources say no".

i have worked on several not-faang-sized-but-largish b2c startups, all of which were forced to spend a very large amount of money, time and people trying to migrate things off the one central RDBMS when it could no longer be scaled vertically. This is a real problem that you likely will run into well before FAANG size, and it really does hurt, and i think it gets minimized by the "just use one postgres for everything" trope. You should probably think about it at least a little if your planned business trajectory is into the 1 million+ users realm.

Im not saying you have to use nosql or whatever, but some sort of small future proofing (eg introducing different schemas/permissions for your different data domains inside your monolith) can help make future database seperation much easier.

If you're doing a B2B app its a different story, much harder to hit the limits of a big RDS instance.

The real answer, as in most software engineering is "it depends"

How long ago did these startups hit the limits of vertical scaling? Those limits are always increasing. Expensify's famous post from 2018 [1] suggests that it should be feasible for one monster machine to handle millions of concurrent users.

[1]: https://blog.expensify.com/2018/01/08/scaling-sqlite-to-4m-q...

theres one that im still working on today :) for practical and all the other usual reasons most people are going to rent cloud dbs (like RDS), not run bare metal. Also, theoretical/benchmark performance is a very long way away from the actually achievable performance when you have 100s of developers actively evolving a codebase against the database. Optimal query usage etc in that scenario is an impossible pipe dream, best you can do is "not crazy", unless you grind all development to a halt behind gatekeeping dba-types.

There are many other operational challenges to single very large database instances. Upgrades, backups, migrations, etc all become way more risky and hard when you get into the mega-db range. The number of total outages ive seen caused by migrations gone awry, query plans gone awry... RDBMS are incredibly complicated beasts and at huge sizes can be very unpredictable in a way that many smaller dbs are not.

Micro-services is about partitioning based on functionality and features -- so I think more often than not, 99% of the microservices use 10% of the storage and 1 microservice use 90% of the storage, and you're back to solving the problem of horizontally scaling your storage regardless of microservice vs monolith.

thats sometimes true, not always. but a good observation. Even if it is true, having that 90% usecase behind its own service often allows a lot more flexibility to scale for that particular storage problem , and not have to use it generically/everywhere. Eg, time-partioning, cold archiving, sharding, whatever is suitable for just that use case.

> The real answer, as in most software engineering is "it depends"

Of course. It's not as if only FAANG has scale issues.

However the set of people who believe they have scale issues is orders of magnitude larger than the set of people who have legitimate scale issues. People should start by assuming they're not in the second category is my point.

There is another aspect to this - if you're working on a startup that will scale, if successful - not to FAANG scale, but let's say many millions of DAUs and thousands of QPS on the servers - you also want to build some kind of future proofing into your architecture.

And then begins the dance of balancing what we currently need (we have maybe 100K DAU and launched recently) and what we predict we might need if things go right. So you don't need to go full on K8S with some crazy sharded database and 50 microservices on day 1, but you also don't want a PHP/MySQL monolith if you're building something that works like, say, Twitter, because you'll have to scrap ALL of that under huge pressure if things start to take off.

> 100K DAU

> but you also don't want a PHP/MySQL monolith if you're building something that works like, say, Twitter

This is a lot of whiplash in regards to discussing size.

I just want to say...

100K DAU with a PHP/MySQL monolith is 100% possible - I'm certain a lot of people here have achieved this without much problem. Things like Nginx, PHP-FPM, reverse-proxy-caching, load balancing to stateless application servers, read replica, offloading static assets to CDN, blah blah... I digress but you can 100% hit 100K DAU with a traditional PHP/MySQL monolith.

Yes, my point is it will hold, but it won't hold 50M users, and if you take off you'll be scrambling to rewrite it from scratch. So a flexible architecture that is simple but can scale (as an architecture) might be a better choice than the simplest possible design you can make.

Could not disagree more. If you are a startup that is struggling to survive, it is an absolute waste of resources to overengineer your app for potential scale. It's the VC way to burn all of the money hiring talented infra engineers to manage this, but it is not pragmatic at all. The exception is if your startup is literally a high volume data processing platform or something similar.

That simple PHP/MySQL or rails monolith will scale much further than you think by throwing more servers at it without having to hire an army of devops engineers. Solving problems you don't currently have when your company is not profitable is a waste of money.

Maybe the monolith shouldn't use PHP and MySQL. But what about, say, a high-performance managed runtime (e.g. JVM or CLR) and SQLite? Expensify's findings [1] suggest that with vertical scaling, SQLite can go very far.

[1]: https://blog.expensify.com/2018/01/08/scaling-sqlite-to-4m-q...

Reply on news.ycombinator.com