I'm just wondering how an infra would look like in a small company, with 1 person dedicated to the infra, and around 10 devs working on a monolith.
I'm guessing that a lot of people will have a lot of different opinions. Just how many possibilities could we see there ? And how would those impact the business and the devs job ?
Use an IaC system (Terraform, Pulumi, etc) to manage everything from Day 1:
1. Use a major cloud provider (AWS or GCP, probably)
2. Get a managed HTTP load balancer (ELB, ALB, whatever)
3. Package your app in a container image, and run your app on 3+ containers behind the load balancer (using on bare VMs, K8S, whatever you prefer), ideally at least 2x containers in 2x AZs.
4. Set up a managed database cluster with Postgres or MySQL and run it with multi-AZ and failover
5. Run 2x VM instances (for redundancy) for asynchronous jobs (using a message bus service or using your database as a work queue), ideally 1x in each of the AZs your database is in
6. Store any large files in cloud storage and put them behind a CDN
That's all 99% of companies will ever need to do. These are all old technologies that Just Work.
If you use a container orchestration service (ECS, GKE) then you use that system to upgrade container image with new app versions.
If you use autoscaling groups / managed instance groups, then you use that to replace VMs with new ones running containers with the new app image.
Using rolling updates, the load balancer drains connections to a container and then it is replaced.
Ideally this is done using your IaC system. So a deployment just involves changing the container image (app version) and applying the update.
Then there's more nuanced things that most teams will miss early on without someone pointing the problem and the solution e.g. decouple configuration from the app, design stateless apps (e.g. 12 factor app), use secrets management easily (e.g. dynamoDB based solutions like credstash are dirt cheap, AWS secrets is okay-ish), used managed DBs (RDS is the most common choice) and more.
I have provided infrastructure-as-a-service for many many baremetal/VMS/containers, but then I wasn't managing them (e.g: it was many small team's infra, not a small team of 10 people using 20-120 VMs as per OP)
The number of nodes was used to run a monolith. Same deployments and all, wasn’t as complicated as it might sound (terraform to manage infra, few custom apps to talk to external APIs, etc)
These were definitely not 500 fortune companies but AFAIK all of them had some cash flow or were acquired by behemoths.
Very refreshing to read in this day and age where it’s not uncommon to have more micro services than paying customers.
This has worked for years with practically zero maintenance.
> This has worked for years with zero maintenance.
Do you apply security updates, e.g. for Caddy, the OS, db, etc.?
And that's when things are working as expected. All it takes is for one of your non-DB services to be compromised, and an attacker can now connect to the DB on localhost. That's why it's best practice to put a secure password on your DB even if you only expect connections from local services. And yes, you should upgrade it too... or at least apply security patches.
I'm guessing you're OK with the tradeoff of your machine going down sometimes?
Sure, there's downtime when the VPS provider is down. That's no different than "managed" services.
Do you have some kind of monitoring? What about logging?
Crashes are logged but I don't think anybody has ever looked at the logs. I'm not sure a crash has ever happened (the app is written in Haskell). Unexpected errors are saved to the database and can be viewed using the web. If we needed logs people could tail them from the VPS (ssh -t $WEB_HOST 'tail -f webserver.error.log'). I know, I know, that's crazy. I'm told real software companies are supposed to ship them to someone else, and then pay them to view the logs using a crappy web interface.
No monitoring. If the site is down customers will let us know, which has happened a few times. Monitoring wouldn't help much with shipping bugs anyway.
For a monolith and size you are talking about a sensible approach would be pretty minimal infra. The app would be hosted on a managed service (e.g cloud run, elastic beanstalk, app engine etc…) along with some kind of managed database service.
You might also see a kubernetes cluster being used - generally happens if that’s what the original devs knew.
Adventurous places may have the monolith deployed on serverless which can be pretty cost effective.
Heroku like services are also still popular.
In my previous place we had the monolith on cloud run.
Anything more complex than the above would to me be a bad smell - unless there were some very good reasons.
How was managed a new deployment ? Could anyone push a new version ? Did it go through CI/CD ?
And about CI/CD, did you have something vaguely resembling it ? Was it all hosted ?
> Anything more complex than the above would to me be a bad smell - unless there were some very good reasons.
Yes, you would probably not need something more complex in that case, but then again, some companies with 5 devs start with microservices
We could have used googles cloud build, but the team knew GitHub actions well.
In our case that’s a rack in a datacenter, a SAN, MAAS, VMs, terraform, and chef.
Your replacement will thank you, and so will management for a smooth transition.