I actually did use to have a bunch of virtualization and kubernetes and crap, but got rid of it because it ate literally half the systems' resources, in a death by a thousand containers-type of way. There was also a lot of just jank, stuff was constantly breaking and I was always trying to piece together why from half a dozen out-of-date tutorials. Felt like herding cats.
The baremetal server was the easiest to set up, obviously, but the hardest to maintain. Where did I put that random config? Wait, why did this break when I upgraded X? Switching to Docker (and later Compose) wasn't that difficult, and made maintaining services much easier. Going to k8s has been challenging, mostly because of how I'm doing it - a controlplane and worker node on each physical node in Proxmox. Writing the YAML is, well YAML.
I'm mostly looking forward to not having to keep my VMs updated. I automated a lot of that (VMs are Packer templates) with Ansible, but it's still annoying. Upgrading k8s nodes is a lot easier, IMO.
Maybe I don’t know what I don’t know, but my setup works for me and I don’t really have any problems maintaining it so I figure why add all the complexity?
It always feels weird to see threads and threads of people talking about dozens of software programs I’ve never even heard of, let alone used. Maybe I’m living in the past but to me a “stack” is: OS, server, database, application. Like LAMP. Wonder when this changed!
It makes me curious about what kinds of stuff people do in their home networks that I never even considered doing.
It is just a learning project, for me K8S didn't 'click' untill i tried to configure it myself on a few machines.
There is something about doing it om physical real things that aids learning, like you could read all chemistry textbooks in the world but untill you actually try it yourself it's not quite the same
I think it's important to keep you 'precious files/services' and you experiments separate.
For docker there's a simple motivating case: some services are difficult to configure securely with minimal permissions, and having a standard docker image provided by people who know what they're doing would be a big net win for security on the internet. There are a lot of poorly configured and insecure http server's out there. Think about how many vulnerable http servers are running on routers.
I think the number of router vulnerabilities alone refutes this argument. Yes, security wasn't the original motivation, but it could be one good motivation.
At home I just want things to work, hence a nice simple LAMP
The number of times I have to do something to the 160 machines I'm responsible for is about 3 times a year. Clusterssh does the job in 10 minutes, and inevitably shines a light on some unexpected error somewhere. If I was doing it on a weekly basis though, using ansible would be beneficial.
Personally I don't like ansible's way of throwing everything in a file in /tmp, then running it as root. authlog gets sent to my central syslogs servers so I've got a record of what happened. With ansible I'm never quite sure, and I'd have to look in two different places for logs.
Instead we just have a lightweight phone home agent which reports processes/diskspace/mounts/interfaces/hostname/ownership etc so you can identify who owns a device on your network and what it's responsible for without consulting out of date documentation that relies on humans to do things.
Kubernetes I guess is useful if you're building a cluster of 100 machines scaling and doing the same service because you're handling a million users a day. Most companies don't do that, most services don't need that.
Use the right tool for the right job, and don't assume the stuff google use is the right tool (unless you're building a lab because you want experience to get a job working for google. If you tried that shit on my network you'd be starting your job search far earlier)
Why do you actually want this? Even having one ES is a bit of a smell that you're probably over-engineering things.
If it's just a single server, you're fine with logrotate and grep.
Who said graylog is only logging on that single server? That server is by far the biggest and hence why I run multiple services with big dependencies on it, but I have that server, 2 cloud servers (one running my personal web page and other publically accessible services, one running a couple of game servers) + a couple of Raspberry Pis.
Even if I didn't have all that, a web interface to configure it to email on ssh login or send me a mobile notification that my offsite backups are still happening and/or not happening for devices or similar items would still be a nice feature, even if you personally feel it's too complex or you could write a cron job for each of them.
1. Unless you have crazy dynamic scaling needs, there's literally no point in using k8s or nomad 2. Unless you want RBAC and fine tuned ACL, theres literally no point in using k8s or Nomad
I hate this idea that you need a Ceph cluster and a 3 node HA cluster to have a home lab.
Not saying don't do it because you want to play/test/learn any of those platforms, just that if you are going to run a server service maybe consider if you are doing it for learning a task orchestration service or just want to run XYZ reliably.
Agree that Ceph is ridiculously overkill though, I tried. Longhorn is much easier and perfectly fine for distributed storage.
I'd suggest if you already know Ansible then don't change. If you have no idea which is which then try them both and let us know :)
The reasons that I used it were because the agent (called “minion”) initiates a connection towards the master, which can be handy if you’re behind NAT- but it was interesting to us because incoming connections are easier to manage at scale than outgoing ones.
Another reason was that the windows support was much more mature (though not perfect) and our environment was mostly Windows servers.
That said, installing the minion agent was easy, much easier than enabling winrm.
If you have sunk significant time into ansible I wouldn’t recommend switching, but it’s definitely not dead as per the sibling comment, I personally found it more enjoyable and easier to work with once understood the DSL pattern and added a few custom modules, it’s very simple underneath.
I don’t want to build a good chunk of Kubernetes myself just to host my software. It’s nice to be able to Google for solutions to my problems or to ask on various forums/chat rooms (or in many cases, not have the problems in the first place because someone else committed the fixes to Kubernetes long ago).
Now I'm building a small platform to host a business I've started and I'm going back to ASGs with simple EC2 Instances.
There's really no need for the features Nomad/K8s offer at almost any scale outside of a few big companies.
Also, I'm curious why you didn't just end up sticking with Nomad now that you know it.
If you're going DIY, however, then you're not doing yourself any favours by starting out with an orchestration engine.
In the case of Nomad you'll also need Consul for it to be truly useful. That means six EC2 Instances (for example) for a truly production trade system - 3x for Nomad, 3x for Consul. Of course you also need mTLS as well - may as well throw Vault into the mix too, then... that's another three EC2 Instances to support that cluster... 9x EC2 Instances into the DIY solution and all you have is an orchestration engine and you've not even started on the workers yet. That cost so far is USD$30,416.04 for the reference architecture for all nine nodes: https://learn.hashicorp.com/tutorials/consul/reference-archi...
On the other hand if you keep it simple and just work with AMIs you bake your self when you need to release something new, you need two EC2 Instances. The cost of the compute resources for this for 12 months: UDSD$330.
There's no raft consensus which needs three nodes to be stable (but a single AZ failure can result in split brain, so really you need five AZs), no new technologies to understand, no new vendor to work with or pay capital to, nothing more to support.
These things - these orchestration engines - have their place, and that place is outside of 99.9% of businesses (large or small.)
As for moving away from Nomad: too expensive and too complex. I ran Consul, Nomad and Vault on the same master instances (which is a risk and against best practice) without mTLS (just TLS) and it took a week to get it all running. None of that includes monitoring, backups, auto-scaling, etc.
I believe the work involved with going DIY on the orchestration front is not worth the benefits in the short to medium term. In the long term you'll likely outgrow your own simpler solution and at which point you'll likely have the capital to switch to a managed service anyway /shrug
It's all relative, after all.
The beauty with HashiStack is you don't need to go fully monty. There's also no need to use ACLs which I agree adds a lot of unnecessary complexity if you're only at the MVP phase or doing things solo/in a small team. And you can run everything on a Digital Ocean $10/mo droplet -- it's very lightweight. I even have one HashiStack deployed on a Raspberry Pi running Home Assistant and other ancillary wares and it works great. But there's always the path to building a proper cluster if you need the redundancy.
I'm very bullish on HashiCorp. They can do a much better job though of communicating that not every deployment needs to be enterprise-grade and offer better ways of bootstrapping the type of cluster that fits your needs. I think a lot of that messaging gets lost due to their ongoing war with Kubernetes and what large companies are looking for.
I think if you're at the MVP/solo-team stage then a single server that you push the app too with Ansible/Puppet Bolt, run via systemd, and then monitor, is even simpler, easier and "cheaper" (in time, which is money) than having to learn and use the HashiStack.
My app compiles into a single binary thanks to Go's binary packing feature. Does deploying a single binary reeeeaaallly need the entire HashiStack? Seriously? Do I need TWO operating systems (the host's and the container's) to run a binary?
Honestly I just think people over think this stuff and are easily swayed by cool features.
I guess that's fine, I'm just used to having labs to test how I can avoid single instance failures. :)
I've had great success with Ansible+Proxmox. The proxmox CLI interface is good enough that you can basically do DIY terraform like setup straight against CLI without any agents / plugins etc. (I know declarative vs imperative but hopefully the comparison makes sense)
Lately I've been toying with local gitlab-CI building and deploying images directly into the environment so that any changes to the code redeploy it (to prod...this is home after all haha)
Considered terraform too but not a big fan of their mystery 3rd party plugin provider system
Sadly, I leaned this the hard way by setting up a 4-node k8s cluster and realizing bare-metal support is abysmal and the maintainers simply don’t care about you if you’re not running a cluster on AWS or GCP. Unfortunately my day job still involves k8s and people give me strange looks when I suggest metal so substantially less complex, cheaper, more performant, and easier to orchestrate and maintain.
A few months back I built a Nomad based cluster with Consul and was like: wow! This is super cool... but it's also expensive and complex.
I'm going to be building out the platform with an ALB, two EC2 Instances in an AutoScaling Group based on an AMI I bake with Ansible and Packer. I'll use Terraform for the entire infrastructure.
The release pipeline for content updates (it's an MkDocs/Material based book with videos blended in) will be me updating the AMI and firing off a `terraform apply`.
I won't be using Docker. I won't require or need an orchestration engine.
It's a static website delivered via a custom web server written in Go (as I need custom "Have you paid to view this content?" logic.) That's about as complex as it gets for now.
I'm actually looking forward to implementing it and watching as something so simple hopefully goes on to allow me to build and scale a business.
k8s and nomad are on my "maybe eventually later" list though.