Hacker News Reader

Comment by jimcsharp | Hacker News Reader

I wonder when this type of work will go away. I see it everywhere.

it will never. databases are hard and people are idiots. nevermind something as hard as SQL optimization and index design -- there are LOTS of people out there who think creating a new logical database for every new customer is a good idea, resulting in 2000+ databases on a system when there should be 4. people who think HA is a waste of money because nothing has ever failed on them. cheapskates who will under-spec deployments and then yell at you when it doesn't fit their data. people are stupid. do not underestimate the stupidity of your average tech industry "professional". this is why i think there should be a traditional engineering certification process for this stuff.

also, i simply don't buy the assertion that any ops or dev can be automated in any meaningful way. the only thing it's doing is making it easier for morons to shoot themselves in the foot by ignoring the experts and then run to the same experts like a little child when everything catches on fire. see: recent story of a guy who deleted 1500 systems at once. oops.

are we all doing more, or less work than 5 years ago? has all this cloud and deployment tech made any less work for anyone, or destroyed any jobs? lol no. i repeat emphatically LOL NO. such a claim is just absurd in my view. anyone who claims such a nonsensical statement has never done actual ops work where you get called if shit goes south.

the smartest thing amazon ever did was simply give people a button to push to give them more money when they're faced with their own overwhelming stupidity. ain't nobody to blame but yo'self then.

> there are LOTS of people out there who think creating a new logical database for every new customer is a good idea, resulting in 2000+ databases on a system when there should be 4.

I'm genuinely curious as to why you feel this way?

see? LOTS.

An ungodly amount of reasons. The ability to manage that many, for one. Those poor devops folk who have to deal with that.

On a more technical side as an example, the number of open files can only be so high. You really start to hit the limit when you do things like that, which is when the most interesting kinds application crashes happen. "Just increase the file count." results in "Our report shows that an OldServer did not include a necessary open file limit config change. Engineer #582 has been fired."

CLI tooling has solved a lot of the problems that arise from many. Scripts for migration, host transfer, creation and deletion took some time to build out, but do work fine.

I suppose we are lucky to have a very consistent workload per logical database.

This allowed us to calculate our costs ahead of time. We found that we could keep DB costs at ~$1/customer/month. For a B2B SaaS product we were satisfied with that cost.

My personal preference for a single multi-tenanted schema as opposed to 2000+ schemas is: Unless you require per customer customisations (which is a fast way to ensure you have an upper limit on the number of customers you can support) there is no need to duplicate. Use a single schema where every table with per-customer data has a tenant id column.

Most decent RDBMS have a good way to ensure isolated locks across tenant ids e.g. MySQL partitions.

Understood.

Currently, I manage several hundred logical databases for a B2B SaaS product. This has the potential to grow to several thousand over the course of years.

My customers enjoy that (arbitrary) piece of mind that their data is logically separated from all else. I enjoy not having to remember to include a tenant_id clause.

Our traffic pattern is quite steady per hour of day and per customer. That, plus using RDS for horizontal scaling (fixed # of logical DBs per host) allowed us to take this route.

I suppose the decision is a matter of circumstance and preference, but I don't think this is necessarily "wrong" to do.

it's going to work fine until it doesn't. that's not going to be a good day. smart people can rationalize anything.

On the flip side, learning resources will continue to improve. Now days all one would have to do is google "database optimizations" and immediately see things like indices etc. Furthermore if they're using a framework, which most are, the search becomes "how to speed up django" which leads to a lovely documented page hand holding you through whatever you need to do.

When companies figure out that hiring "one guy that can do everything by himself!" is a bad idea. So...... not likely anytime soon.

Except... sometimes one guy really can do everything by himself, and do it much better than the other folks. However, usually the person who can do that is "too expensive" and they'd rather spread the cost (or a higher cost) over 3-4 people.

It's not just talent/skills - there's more communication overhead when dealing with a team, and when the team has a range of skill levels on it, the comm overhead can make things worse (if not managed well)

I have seen the one guy replaced by a team of 5. The team got less done than the guy.

I have a longstanding suspicion that developer cost climbs slower than developer skill and productivity, meaning companies should always hire the best they can get, because they'll maximize productivity that way.

IME, it climbs WAY slower.

I am the one guy fixing these things all by myself.

Reply on news.ycombinator.com