Single RDMS database + stateless middle tier is still best solution for 99% of applications out there.
But even fighting for RDMS in 2022 is getting daunting as so many people will immediately start pushing for document DB's, GraphQL, etc. Seriously - I feel like SWE's have no idea the powerhouse a modest MySQL/Postgres instance can crank out while perfectly maintaining their data (ACID, replication, etc).
Everyone wants to use tools that were designed for FAANG scale regardless of org size. Anecdotally I've fought this hard because it's almost always at incredible detriment to the business's needs.
I find that modern devs balk at "single RDMS database + stateless middle tier" but only for selfish reasons like resume-driven development. This pattern is really burning me out more than anything else of my career.
Again, all this is personal anecdotes.
So they'll see some numbers that are on the upper end or beyond what they're used to seeing, and to them that's big scale. How do you solve that? Well, you reach for the tools other people have used when they have "big scale". All the literature says you can't just increase your instance size when you have big scale, so why would you try that?
And yes, that's wrong. But it drives a lot of the mentality.
Seriously, you can launch a server that has almost a terabyte of RAM! Is your business doing enough to make that database even blink? "My sources say no".
And then begins the dance of balancing what we currently need (we have maybe 100K DAU and launched recently) and what we predict we might need if things go right. So you don't need to go full on K8S with some crazy sharded database and 50 microservices on day 1, but you also don't want a PHP/MySQL monolith if you're building something that works like, say, Twitter, because you'll have to scrap ALL of that under huge pressure if things start to take off.
> but you also don't want a PHP/MySQL monolith if you're building something that works like, say, Twitter
This is a lot of whiplash in regards to discussing size.
I just want to say...
100K DAU with a PHP/MySQL monolith is 100% possible - I'm certain a lot of people here have achieved this without much problem. Things like Nginx, PHP-FPM, reverse-proxy-caching, load balancing to stateless application servers, read replica, offloading static assets to CDN, blah blah... I digress but you can 100% hit 100K DAU with a traditional PHP/MySQL monolith.
That simple PHP/MySQL or rails monolith will scale much further than you think by throwing more servers at it without having to hire an army of devops engineers. Solving problems you don't currently have when your company is not profitable is a waste of money.
[1]: https://blog.expensify.com/2018/01/08/scaling-sqlite-to-4m-q...
Im not saying you have to use nosql or whatever, but some sort of small future proofing (eg introducing different schemas/permissions for your different data domains inside your monolith) can help make future database seperation much easier.
If you're doing a B2B app its a different story, much harder to hit the limits of a big RDS instance.
The real answer, as in most software engineering is "it depends"
[1]: https://blog.expensify.com/2018/01/08/scaling-sqlite-to-4m-q...
There are many other operational challenges to single very large database instances. Upgrades, backups, migrations, etc all become way more risky and hard when you get into the mega-db range. The number of total outages ive seen caused by migrations gone awry, query plans gone awry... RDBMS are incredibly complicated beasts and at huge sizes can be very unpredictable in a way that many smaller dbs are not.
Of course. It's not as if only FAANG has scale issues.
However the set of people who believe they have scale issues is orders of magnitude larger than the set of people who have legitimate scale issues. People should start by assuming they're not in the second category is my point.