If you create difficult solutions, you need to create a lot of documentation. And there is nothing that people hate more, than writing documentation :).
If you create difficult solutions, you need to create a lot of documentation. And there is nothing that people hate more, than writing documentation :).
„But we can scale from 500k to 50 million users now“, which usually doesn’t happen. But what does happen is, that the system gets too complicated and too hard to extend or fix bugs.
1.) No one writes blog posts about using old boring stuff and meeting expectations. And such blog posts can contribute to recruiting and even, indirectly, sales, by helping your company get general press for being "on the cutting edge."
2.) You can potentially get better overall results even if the new tech has more gotchas and unknowns, because certain developers will knock themselves out to deliver on time and prove you were right in trusting them with the platform/language decisions. But you have to pick very wisely on this and keep in mind such devs may immediately lever their experience on the new tech to get a new job and leave you with a team of juniors that wander lost without their guru.
One thing I found useful is: try to build something that is boring old tech, but the boring tech can be switched out easily. Like a monolith that can be also split up into micro services (via a build flag).
Typescript and Golang for example, are relatively new-ish but still solid and boring, and have significantly much less gotchas than older established tech like, say, Rails.
I don't do Java, but I see it getting a bit of a renaissance. Either because of new features or because people are moving to Kotlin (yep for backends), which is a different language but still has the "solid boring" feel to it.
It is tough to hire for devops, though. The currently experienced people want to work with more complex infrastructure. Maybe we need a few more years to see those people getting burned and deciding they don't want to life an exciting work-life anymore.
I'm also one who uses boring old technology. Mostly relational databases and LTS operating systems. May not make for interesting blog posts but I'll take that over being called at 3:00AM because some microservice isn't working.
So I tend to build a system that can handle the load that is expected, and some reasonable scaling (maybe 5x to 50x more load).
If your product grows way faster then expected (1000x), you will probably have way more budget, and be able to do it even better.
Also, unlike sales/PMs ideas (well, at least a few of them), those complications created by developers aren't really helping sell the software. Nobody is buying software simply because it has 50 microservices, all the design patterns and needs Kubernetes plus a few thousand-bucks per month of cloud to run.
If only sales and dev drive product direction, with neither of them having explicit focus on what the end user needs, you end up with overly complicated, half-assed products, that kind of work, but not really, and nobody wants to touch.
On the other hand, what matters is money in the bank, so doing stupid shit like this is fine if the cash flow looks good.
It's usually the developers telling the sales guy "Sorry man we can't deliver this feature on time because our Elastic Search cluster is on Fire All The Time and we need to fix it before we can start adding new features".
"No, a junior can't add a filter to this column because the current system doesn't support filtering aggregates, so we have to partially rewrite it. No, we can't add this directly to the controller or model because there's no controllers or models anymore! Or more like: there's just one! Why? Because we wanted a future-proof system and decided to make a GraphQL simulacrum that allows adding new features without almost no cost! Sure, almost nobody understood it and sometimes it requires rewriting, but who wants to working on a fucking CRUD system?"
You could not make sophisticated HTTP-requests, because you had to use the in-house library. And off course it had much less functionality than some commonly used libraries.
So what I learned from that: abstractions can be nice, but they are useless if you can’t escape them. They must have „escape routes“ everywhere, to do things that are not covered by the abstractions.
There's unfortunately no known way to make good software other than keeping it simple, concise and without too many components.
The thing you can do to prevent it: continuous refactoring. It’s better to ship features on time and clean up afterwards, than cleaning up a lot before and never ship things.
As I said, using escape routes for some time, for speed, is acceptable. The problem is making this into a rule.
Refactoring a bad abstraction is easier than refactoring an abstraction that is messy and bad. So either refactor sooner (and create proper abstractions) or skip the “ball of gold” altogether (eg: use the original framework directly).
That still implies writing to an audience other than yourself, and the Curse of Knowledge still applies.
In many ways, it's much worse, because it's easy to assume that because the document is for a technical audience that the reader will know as much and have the same context as the author, and easy to include less supporting material. I look at StackOverflow and see how many questions are technically answered in the documentation, but in fact do not prevent the question from coming up. It's not the the questioner can't understand or the question is stupid, it's that the mental model of the reader and writer don't match up.
Of course it's much worse if, as in your example, the knowledge is in one person's head. That's a related problem, and having something is better than nothing. Even with some documents, cases still arise where there's a missing piece of information that the document author didn't record. The author understood what the document meant, not realizing it is only understandable given some knowledge the author did not provide.
The entire point of writing software is to communicate complex ideas to others.
Most programers are no good at writing code the communicates to other humans. They write code that computers can understand, which is a completely different skill.
Yes, there is - keeping huge documentation up-to-date.