ghomem

The questions are short but the answers would be long. Puppet manages all fine grained OS resources (files, dirs, repos, cronjobs, sudo declarations, firewall rules, etc) and you aggregate those resources into classes which are then pushed to different machines. The classes are parametrizable for the differences between systems.

If I was to write an idempotent script for each native resource I would finish in some years :-)

You chose whatever monitoring system you like the most.

For offline nodes you use whatever the level of criticity of your node justifies. This is something people struggle to understand: not every business needs 99.99% uptime. That said, I never had a downtime in Hetzner. On Digital ocean I had one short forced reboot in 4 years. YMMV so protect yourself as much as necessary.

Deploying on a different provider than Hetzner is the same as deploying on Hetzner except the part of launching the machine which is trivial to script - the added value is making the machine work and Ubuntu/Debian/RHEL are the same everywhere. You don't have vendor lock in with this.

If K8s works for you, enjoy it. Nobody is telling you to stop :-)

I don't mind elaborating - the fact that people are asking me questions reminds me that I need to invest a bit more effort on some articles.

This case is actually pretty simple.

Puppet applies the configuration you declare impotently when you run the Puppet agent: whatever is not configured gets configured, whatever is already configured remains the same.

If there is an error the return code of the Puppet agent is different from that of the situations above.

Knowing this you can choose triggering the Puppet agent runs remotely from a monitoring system, (instead of periodical local runs), collecting the exit code and monitoring the status of that exit code inside the monitoring system.

Therefore, instead of having an agent that runs silently leaving you logs to parse, you have a green light / red light system in regards to the compliance of a machine with its manifesto. If somebody broke the machine leaving it in an unconfigurable state or if someone broke its manifesto during configuration maintenance you will soon get a red light and the corresponding notifications.

This is active configuration management rather than what people usually call provisioning.

Of course you need an SSH connection for this execution and with that you need hardened SSH config, whitelisting, dedicated unpriviledged user for monitoring, exceptional finegrained sudo cases, etc. Not rocket science.

Once you do it for long enough it might be worth it to consider configuration management where you declare native structured resources (users, firewall rules, nginx reverse proxies, etc) rather than writing them in shell.

I use Puppet for distribution of users, firewall rules, SSH hardening + whitelisting, nginx config (rev proxy, static server, etc), Let's Encrypt certs management + renewal + distribution, PostgreSQL config, etc.

The profit from this is huge once you have say 20-30 machines instead of 2-3, user lifecycle in the team that needs to be managed, etc. But the time investment is not trivial - for a couple of machines it is not worth it.

Well that's exactly the point! Creating complex cloud resources with, for instance, Terraform, is less reproducible than a shell script on an LTS system like Ubuntu or RHEL - that's because the cloud provider interfaces drifts and from time to time stops accepting the terraform manifests that previously worked. And to fix it, you have to interrupt your normal work for yet another unplanned intervention in the terraform code - this happened to my teams several times.

This does not happen with Puppet + Linux, because LTS distributions have a long release cycle where compatibility is not broken.

I tried to explain this topic in the article linked above. Not sure how far I succeeded.

I had the exact same business for 18 years :-) and yes, without corporate nonsense it is easy to retain intelligent people. Cheers

My comment about pip is orthogonal to Docker. This is the same with or without Docker - I added a comment on this thread with more detail.

If you really want to open that can of worms, here it goes:

Pipy is an informal source of software that has low security levels and was infested with malware many times over the years. It does not provide security updates: it provides updates that might include security-related changes as well as functional changes. Whenever you update a package from there, there is a chain reaction of dependency updates that insert untested code in your product.

Due to this, I prefer to target an LTS platform (Ubuntu LTS, Debian, RHEL...) and adapt to whatever python environment exists there, enjoying the fact that I can blindly update a package due to security (ex: Django) without worrying that it will be a new version which could break my app. *

Furthermore, with Ubuntu I can get a formal contract with Canonical without changing anything on my setup, and with RHEL it comes built-in with the subscription. Last time I checked Canonical's security team was around 30pax (whereas Pipy recently hired their first security engineer). These things provide supply-chain peace of mind to whoever consumes the software, not only to who maintains it.

I really need to write an article about this.

* exceptions apply, context is king

I'll correct myself:

s/host OS independence/a certain level of host OS independence

And getting containers to run depends on the OS - if you don't control the host, leads to major ping-pongs.

Even within Linux (Ubuntu, Debian, RHEL, etc) when you are distributing multiple related containers there are details to care about, not about the container itself but about the base OS configuration. It's not magic.

I fully agree with you: it is mostly an education problem and you can find people willing to learn right out of univ. Indeed, that is exactly my experience: I successfully onboarded several (carefully selected) junior people into the ops skillset over the years and I have seen them do wonders with customer systems, while enjoying their "ops life", without having fires every day.

The connection of this to the replies above it: I am not sure if this kind of junior people would be easy to retain in a large corporate environment. We certainly can do that in niche consulting.

I mentioned Docker because it interests many developers but on VMs that I control I do not need Docker at all. Deploying with Docker provides host OS independence which is nice if you are distributing but unnecessary if the host is yours, running a fixed OS.

For Python backends I often deploy the code directly with a Puppet resource called VcsRepo which basically places a certain tag of a certain repo on a certain filesystem location. And I also package the systemd scripts for easy start/stop/restart. You can do this with other config management tools, via bash or by hand, depending on how many systems you manage.

What bothers me with your question is Pip :-) But perhaps that is off topic...?

The same thing that happened to devops from 2017-2024 (see: https://logical.li/blog/devops/) is happening with dataops. Hype train and jargon based decisions are taking place.

In the past years I was solving a data pipeline mess on a project which also had a devops AWS mess. First thing I was told was "what we need is a data lake".

Decisions are sticky so take context into account.

Reference:

https://survey.stackoverflow.co/2022/#developer-profile-deve...

Only around 11% of the whole devs identify as devops specialist or cloud infrastructure engineer.

This is why I am saying ops people are a scarce good (unfortunately) from a data driven perspective. Of course my daily life confirms it.

Probably so. And that would be likely my approach at such scale.

Still, my most benevolent interpretation of current reality is, rather than saying "that cloud native stuff is crap", accepting that there are cases where it may make sense.

For instance, large companies might have trouble hiring a good ops team because they have in general trouble hiring and retaining talent (another conversation topic).

Ops people are a scarce good because univs do not train people for that and most people prefer coding. I am leaving the work devops out because the market completely perverted its meaning.

(my take on the devops funeral: https://logical.li/blog/devops/ )

Fake dichotomy. It is not old vs new, it is simple vs complex. The fact the older technology is simpler is just a coincidence.

...and the funny thing is that it is still cheaper than cloud native even being up all the time and provides a predictable cost per month, unlike serverless where you can have big surprises.

Check:

https://logical.li/blog/emperors-new-clouds/

And if one of those services is down, your entire application is down. You basically build a server made of abstract components ECR, S3, ALB, CF , all of which are able to fail.

... and this!

More of this.

hahahahahah that was funny :-)

> There was a team of 5 people but something like 10-15 services

Puff! Talk about microservices! Or is it macropeople?! :-)

I am not against the cloud. VMs are also cloud, unless you run them on your own servers. For instance, the Hetzner Cloud (mostly VMs, plus load balancers and disks) is so cheap and has such a nice CLI API that it competes aggressively with dedicated servers - I would definitely start any with VMs, not with iron.

The biggest problem is the so called cloud native stuff which is both more expensive and more complex. There are contexts where it makes sense but for startups they are doing more harm than good.

I went through sweat and tears with this on different projects. People wanting to be cool because they use hype-train-tech ending up doing things of unbelievably bad quality because "hey, we are not that many in the team" but "hey, we need infinite scalability". Teams immature to the point of not understanding what LTS means have decided that they needed Kubernetes because yes. I could go on.

I currently have distilled, compact Puppet code to create a hardened VM of any size on any provider that can run one more more Docker services or run directly a python backend, or serve static files. With this I create a service on a Hetzner VM in 5 minutes whether the VM has 2 cores or 48 cores and control the configuration in source controlled manifests while monitoring configuration compliance with a custom Naemon plugin. A perfectly reproducible process. The startups kids are meanwhile doing snowflakes in the cloud spending many KEUR per month to have something that is worse than what devops pioneers were able to do in 2017. And the stakeholders are paying for this ship.

I wrote a more structured opinion piece about this, called The Emperor's New clouds:

https://logical.li/blog/emperors-new-clouds/

Update: this is a very nice suggestion. Thanks!

No, actually the only I used was meconet.de which closed for most of the business. That is why I am looking for options.

Thanks for the suggestion!

Typo on https://doesitscale.dev/auth

-> "Authenicatication Services"

Would be interesting (and tough) to compare VMs and containers across providers.

OMG, a reasonable person on the Internet. Nice to meet you neuronic :-)

Judging by the results it seems mature and professional enough to me, including the meme based marketing. And the mission is right too.

Here:

https://twitter.com/htmx_org

But beware of the deep layers of nerd humour.

This is a toy but....

https://github.com/ghomem/definitive-web

https://definitive.li

This was funny :-)