I'm going to scale my foot up your ass (2008)
widgetsandshit.com
widgetsandshit.com
I'm still convinced we could run our entire software platform off 5 EC2 instances and an S3 bucket. But the last 10 years of technical mismanagement turned this into a microservices shit show. Splunk gets more traffic than our clients do.
of unzipped data
Do you really need those 5 nines?
If my bank can go down every week for patching, you probably can too.
What matters if they will pay someone else for that reliability they don't need.
The common denominator of those companies is they all had no significant traffic, had their initial codebase developed by (unmanaged) amateurs, and were later managed by people inexperienced in software.
Due to the heavy resistance in all cases, I eventually became tired of proposing simpler solutions and the endless discussions around it, and accepted premature microservice architectures to be part of my infinite source of income.
We ended up with 3 service. Exact same API is provided. It does not matter : nobody is using it anyway !
Have also seen this coupled with the microservices pattern of having all the microservices share a large common library, with all microservices built together and deployed together all the time. All the pain of shared code across teams plus the pain of shared DB plus the pain of debugging across completely arbitrary process boundaries.
Basically, get boxes on the diagram for each aspect the exec might want to talk about or answer questions about. So that they'd have something to point to, and investors would have some level of understanding, as well as confidence that we thought of and were doing something about each thing. (This is a bit different than the diagrams we were using to discuss and build it internally.)
So I mention to a colleague that I "just know", if a room of tech investors see the Web facet of this (even though that's the easiest technical part of the system), and this one part is all going through only one server (which is all it would ever need), someone is gonna be concerned that we might not know what we're doing about scalability, and I bet the exec doesn't want to get derailed debating that with them.
Colleague advised we go full "System Design Interview!", so I spewed all the unnecessary vapid things we could do, that some semi-technical person might be tuned to look for. Trying to modify the spew to be sensible enough (if encumbering overkill) that, if investors happened to have a hardcore veteran techie vetting (like one of the people who built AWS or Google from nuts and bolts), that they wouldn't become enraged with disgust, and kill me where I stand. The investor meeting seemed to go well.
In a different (real, internal) diagram, for networking architecture, I also broke out out some logical functions into distinct physical servers (some fixed, some variable) in an information/control flow compartment. Then I added a comment on the diagram, something about that these might be combined into a single server for efficiency, for smaller deployments, like customer on-prem. And, I didn't add, if our own data center suddenly needed to host a hugely-popular service for others, and we suddenly had enough hardware for that, the software would be ready (but we'd still only need one server in that one spot that a System Design Interview book-prepped person would think needed a load balancer and workers).
I've also seen a lot of different setups: Minecraft's backend does absolutely everything right IMO, and their sub-10ms p99.9 response times showed as much as well, and I don't have to tell you Minecraft has quite a few users.
Or another client I won't name, that used microservices, Kafka message queues everywhere and the most complex architecture I've ever seen, all in the name of scalability, couldn't even calculate their p99 response times because they didn't even answer 99% of their requests of their requests (that is, >1% of requests were dropped), at loads of about 5 requests per minute.
Moral of the story: the big picture tech decisions aren't always as important as you might think, implementation details/quality can make or brake any architecture.
HA! I wanted to see "break" (as in fracture) any architecture, but I think "brake" (as in device for arresting or preventing the motion) works even better here — the details of the code increase 'friction' in the machine and prevent it's efficient motion...
Overall, I quite agree; the devil is in the details, and those details can swamp the 'architectural' decisions. Best to get both mostly right vs one 'perfect' and be ruined by overlooking the other.
At the time at least, it was very clear that Minecraft was written for single player in mind. Nearly everything ran in a single main loop, causing virtually every multiplayer server to only be able to utilized 1 core. This meant most servers couldn't scale beyond ~50 players with vanilla Minecraft. Community optimizations could take this to 200-300, but usually sacrificed something for those performance gains.
Then a ton of work went into workarounds like Bungee, which enabled the next-gen Minecraft servers to host thousands of players on a "single" logical server. Of course, the main game loop was actually still single threaded and those thousands of players couldn't truly interact with each other as you would expect with vanilla Minecraft.
Getting people to give a shit is totally the harder problem.
Angry dad leaned heavily on the “I’m gonna [X] my foot up your ass!” construction.
I can relate. I had a similar epiphany, but for different reasons.
-Emily
And another which requires 2004 plus
They both require physical access to the same hardware. I want them to coexist on the same physical machine (I can't add a second as I'm space and power constrained), and switch between them trivially and quickly.
Now I haven't actually done this yet, but my feeling is I can dockerise it, push the dev files through, and run one image or another depending on what I need in a given hour.
Full backups are also trivial, if you can afford some downtime: docker-compose down && restic … && docker-compose up. If you cannot afford downtime, the docker volumes go onto a ZFS or LVM partition.
If people want to use Docker without k8s you need to do at least some of what k8s does some other way.
For a single server compose sounds good.
I also imagine there are a few cloud native docker services that do the kubernetying for you, hiding the complexity.
I think you'll find that most devs don't want to have anything to do with operating system management and that docker images extract that away pretty quickly and easily. Just request a VM of a particular size and setup your containers on it. No jacking with terraform or whatever.
I least not lately. 10 years ago : absolutely. But nowadays it’s a docker command or a git pull then yarn/gradle/whatever … and if that does not work something is wrong and a ticket is raised.
Theres already a lotnof complexity in those "simple" parts
From personal experience, I used to use TFS for version control for a few years before using git at my newer job. I also never once had to deal with yarn or gradle.
And I also know from experience, that you can make a unmaintainable mess in every build system.
Also, who is gonna teach the juniors how to use the tools, if everyone sets that knowledge as mandatory? Or do you expect everyone to learn it in their free time before starting any tech job?
The step in those builds should works on developer machines. Is that too much to ask?
If the step in the build do not work? How did that get merged in the first place ?
Ideally, you would
> The step in those builds should works on developer machines. Is that too much to ask?
Quite often, yes. (Secrets that are only on the build server/shared database you cna't connect to without a barely working VPN and such)
> If the step in the build do not work? How did that get merged in the first place ?
Because there is no CI/CD pipeline to stop you?
There are tons of reasons why it can be extremely hard to get into a legacy project as a new person. And that's just setting it up. Ideally, this wouldn't be the case. But sadly, the real world sucks
Last time I worked without reproducible builds was 2005-ish ?
My experience : once someone in the C-suite vicinity enforce a lock-down, things fall in place rather quickly.
Ex : nothing get deploy manually. I don’t care if we miss the next prod push, we will catch the next one.
> the real world sucks
My work is as grounded in the real world as well. And I spend year contracting for bottom feeders company. Hard to do more real crap. At some point is valuable to define priorities.
> Because there is no CI/CD pipeline to stop you? There are tons of reasons why it can be extremely hard to get into a legacy project as a new person
Fine. But lack of validation should not be a reason. If no pipeline checks and no humans checks as well. Then you are deploying un-vetted software.
I assume familiarity with standards tools of the tech stack.
famous words.
It's eerie that I had this experience 5 minutes ago, in a stand-up onboarding meeting for a new colleague: they ran into issues when trying to launch the project through the IDE run configuration (IDE version, plugins, project configuration etc.) but the container based approach that I introduced just in case about a year ago just worked, the first time.
Provided, I actually went the extra mile and documented how to authenticate with the necessary private repos etc., so one could follow a short list of steps bit by bit.
> Lets you test your software in exactly the same environment as in production.
I'd say containers are good for this, but not perfect. Depending on what you're developing and how, you might still run into stupid issues: mostly in setups where you use Windows for your development box and bind mount files, the permissions will get screwed up, as will line endings.
Here's an example: https://blog.kronis.dev/everything%20is%20broken/containers-...
I'd say the best possible developer experience nowadays is along the lines of:
- have GNU/Linux distro X as your developer workstation
- have the same GNU/Linux distro X as the base for your containers (a more cut down version, but with same package repos)
- still ship software in containers, ideally run those on nodes that have some GNU/Linux distro
I can just install some software on the workstation and have it run basically the same as it will in the container, same install instructions and everything (though maybe less focus on clearing up package caches not to bloat container layers, which doesn't matter outside of those). Sometimes that is useful for local debugging or launching things directly with IDE integrations, as opposed to trying to get remote debugging working (though things like DBs, Keycloak, Redis etc. are still better in containers, I'm talking more about back end/front end tooling/runtimes).
If you go to this other post of mine, it actually has an example of this: https://blog.kronis.dev/articles/a-week-of-linux-instead-of-... (search for "Being able to run the same thing inside of containers and on the host system.")
At this point I'm basically unironically shipping my dev box, sans dotfiles. It doesn't scale, but it's the least painful dev and ops experience I've had in years. One can probably take elements from this for more sane and enterprise setups, too (such as focusing on GNU/Linux instead of Windows for development, or at least something with a similar file system).
This is every company older than 7 years whose primary business is not creating beautifully engineered backend software. Unless quality is somehow forced on them by regulation or contracts, then it's just 90% of companies.
It looks tidier, it's presented with the word "docker" everywhere but you're solving the same problems in almost identical ways. Take reverts; you need commits with tags, volumes with backups, etc.
Am I talking about doing it with or without docker there? Well, Both!
This is what you keep discovering - you're doing the same thing in a new way; it's a presentation technology, not a simplification or task reduction one.
Also see the motherfuckingwebsite series, The Best Page in the Universe, etc.
Build your monolith with modules and APIs and split your applications, not your services. If you need to split a module out, your API just has another transmission protocol instead.
As Paul Graham said, do things that don't scale.