Leveling Up My Homelab
cweagans.net
cweagans.net
If something is going to be considered permanent, cut that cable down to length. Either buy shorter moulded cables, or learn to wire cables yourself. Too often have I left a cable long "just in case" only for it to get in the way for years to come.
For patch cables it's easiest and best to buy moulded cables that fit your rack. For things like power cables (extension leads etc.) it's easiest to wire them yourself (at least, in the UK where our plugs are designed to be wired by anyone).
One thing that I haven't found a solution for though: I have a lot of USB and HDMI cable coiled up behind the Beelink boxes (for KVM connectivity). I've found the normal length cables (1', 3', 6', etc), but I haven't been able to find custom length cables for those specific connections. Do you happen to know anywhere I can find those?
The thing that I like about this post is that it touches on many of the difficulties of running a homelab with many physical hosts. You might not need all or most of this setup, but at least you have an idea of how this particular design (a decent one!) scales after reading this.
- Array of off-the-shell compute servers with console access + networking + power
- ArgoCD + GitOps makes a K8 cluster declarative
- Talos makes the physical hosts that provide the K8 cluster be declarative
- Dedicated machines for storage, Control Plan, and Networking isolate the infrequently-changing stateful parts
This homelab does seem compute focused, which might be right for OP but is normally a mistake that people make when they build their first homelab. I'm wondering what OP's home internet bandwidth is. It seems odd to have so much compute behind a network bottleneck unless OP has high-compute-small-output workloads (ml training, data searching/analysis, NOT video encoding)
A single machine with a lot of CPU and a LOT of memory/storage is typically what people want---so that projects they're setting up are fast and having lots of old/idling projects is fine. My old recommendation was a mini-ITX with 128 GB of ram and a modern AMD cpu should take most people very far. Perhaps a smaller NUC/Beelink computer if you're not storage hungry.
However, keep in mind that a single machine will make it hard to tinker with large parts of the stack. It's harder to play with kernel mod settings if you need to constantly reboot all your projects and possibly nuke your lab. It's harder to test podman vs docker if in involves turning off all your contains. A more complicated home-lab gives you more surface area to tinker with. That's both more fun and makes you a better engineer. Of course, you can get most of this experience for far less money if you budget isn't quite as generous.
I personally prefer a digital nomad aesthetic, so I focus on small & simple on-prem hosts paired with cloud stacks. I'm willing to pay a premium on compute to have less commitment and physical footprint. I've been considering setting up a K8 cluster on Hetzner dedicated machines. In my case, that Mini-ITX box is actually a storage-optimized ATX build for backing up my laptop (daily-driver) and media server.
Getting your homelab to boot up and run containers is an honest problem to solve. Figuring out the kernel modules to let you pass through your GPU or run ZFS on root are actual blockers to hosting a gitlab instance.
Running gitlab on multiple nodes to get high-availability is an honest problem to solve. Trying to do it in multiple VMs just to see how it works might teach you something, but it can feel pointless unless it's serving a real goal. I think that choosing a more complicated setup is good because it is hard, and forces you to learn more to achieve the same goal (and ultimately, some of those skills will hopefully be useful).
Additionally, there used to be some limitations on consumer CPUs around nested virtualization, which made it difficult to run VMs in VMs. When I was hosting apps in VMs on a machine, I would want to play around with the machine's configs, but risked disrupting my hosted apps. If I broke something, I didn't have my hosted apps until I got around to fixing my machine. Having multiple machines ensured that I could tinker with one while relying on the other. The process, by accident, gave me an intuition for planning upgrades to infrastructure. This intuition bubbles back into the design process.
I don't often own infrastructure for multiple upgrade cycles professionally, so it is a good way to earn some wisdom on my own.
I kept waiting for the description of what it would be used for, but there was only a passing reference to learning how to run AI workloads.
For some people, buying and assembling hardware is the hobby. This gets old fast, especially when you add up how much was spent on hardware that's now sitting idle while it becomes more outdated year over year.
I agree that for typical learning cases the best solution is a single, cheap consumer CPU paired with a lot of RAM. For the $8000 spent on those 8 mini PCs, you could build a 256GB RAM box with 2 or even 3 nVidia 5090 GPUs and be in a different league of performance. It's also much easier to resell big nVidia consumer GPUs and recoup some of your money for the next upgrade.
It does look fun to assemble all of this into a rack and make it all work together. However, it's an extremely expensive means to an end. If you just want to experiment with distributed systems you can pair 128GB of RAM with a 16-core consumer CPU and run dozens or even 100 small VMs without issue. If you want to do GPU work you can even use PCIe passthrough to assign GPUs to VMs.
Future posts will address some of this. :)
Yep! Right around the $13.5k mark.
> This homelab does seem compute focused, which might be right for OP but is normally a mistake that people make when they build their first homelab.
Very compute focused for the specific projects that I intend to work on.
> I'm wondering what OP's home internet bandwidth is. It seems odd to have so much compute behind a network bottleneck unless OP has high-compute-small-output workloads (ml training, data searching/analysis, NOT video encoding)
1gbps symmetric + a failover Starlink connection. Not a ton of large output workloads at the moment.
> However, keep in mind that a single machine will make it hard to tinker with large parts of the stack.
Very much in agreement here. This is one of the reasons I went with multiple machines.
> I'm willing to pay a premium on compute to have less commitment and physical footprint.
I also like this mindset, but my other hobbies are piano (and I'm very sensitive to the way that the keys are weighted, so I prefer playing a real grand piano vs a portable/mini electronic piano) and woodworking (even more bulky equipment), so I'm already pretty committed to putting down roots.
With the way Ubiquity has treated their software stack on the network side in the past years (major bugs, regressions, and updates that had to be reissued multiple times), I wouldn't trust them with all my data. Ubiquiti's QA was outsourced to the customers and a NAS is the last place where I want to risk bad updates, no matter how many backups I have.
e.g. I wanted to serve tftp directly from the NAS. I can log in and `apt install tftpd-hpa`, but that package has to be reinstalled every time the NAS updates.
I'll be replacing this in the medium term, but I'm not buying more hardware for a little while lol
Cool, but nothing a single compute machine wouldn't get done with a bunch of VM's if learning and home workload are the focus.
This thing probably idles at a couple hundred watts.
It should be obvious to the reader that this is very much overkill, even for the stated goals of expandability and learning.
Incidentally, this setup does leave me some significant headroom in terms of compute resources, but that's by design.
If it's inference, don't expect great performance or cost-effectiveness.
But if you learn a lot during it, I wish you all the best!
If the goal is to have a lot of something so you can play with many different things, this gets the job done. If the goal is high performance and maximum efficiency for learning or compute, a setup with dozens of smaller computers like this is usually not the optimal choice.
It now only consists of a Intel n100 with a big SSD and 32GB RAM running Proxmox. These China TopTon-boxed with their 5x Intel i226-IV network cards are great and can be passively cooled.
Every night the Proxmox makes a backup onto a Raspberry Pi which runs the Proxmox Backup Server.
Next step will be most likekly a VDSL-modem + something from Ubiquiti as the new Fritz! product portfolio is... a weird mess.
Just reading of the logs:
- Backup duration: 3.21GiB in 36s
- Restoring a Snapshot: feels like <<1 minute
It feels with cloud computing a generation of computer scientists kind of missed out on the experience.
Some services I am interested in are hosting my own RSS feed reader, an ebook library, and a password manager. But I'm always looking for more if there are any suggestions.
Check out https://github.com/awesome-selfhosted/awesome-selfhosted
You can do that on a Raspberry Pi Zero for $15, and for $12 you can get a 128gb microsd card, plenty of storage. It'll take up minimal power and fit in an Altoid tin.
Yet. :)
I’m in the middle of my first homelab journey with just a single mini PC with 8GB of RAM. Excited to one day get to this level.
Peak draw could probably be 2kW for a beefy system so electricity costs could really skyrocket depending on usage patterns.
We have pretty reasonable power rates here (https://www.idahopower.com/accounts-service/understand-your-...), so ~$12-20 per month depending on tier.
I've got two R640's so I can live migrate, and an R720XD with TrueNAS (democratic-csi for K8s persistence). QSFP (40Gb) for TrueNAS / R720XD, and SFP+ (10Gb) for R640's linked to a Brocade ICX 6610.
So I can update the hosts, and K8 nodes with 0 downtime. Do I need it? No, but I learned a lot and had / have fun deploying and maintaining it.
I'm rebuilding my homelab [1] too, actually, but deprioritizing it while I stave off a wee spot o' burnout.
I'm excited to see that you're building on Talos; I am too! I used to use CoreOS back in the day, like 8-9 years ago or smth, on PXE-booted VMWare VMs, and I've always missed that cleanliness.
That's a large part of why I'm rebuilding right now - I based everything around Ansible + Terraform, and that's workable of course but working iteratively on a homelab leaves so much cruft around, can lead to incidental complexity, etc etc etc.
Anyway, I'm pumped to keep reading!
Talos is absolutely incredible. There's a learning curve to it, but it's not as steep as it seems.
I started with Ansible, but found myself getting really annoyed at the amount of YAML I was shoveling around for basic things, so I ended up writing a series of bash scripts that rsync files out to where they need to go, run actions when a particular file changes, etc. Provisioning scripts are bundled up and piped to the remote machine over SSH. It's been pretty nice. I'm thinking about building that out into a separate project somewhere.
I'd love to check out what you're working on! The link seems to be broken though.
> The link seems to be broken though.
Yeah, I'm a world-class infra engineer. smdh. Changed how the DNS record was created but didn't push my changes so they were reverted by a scheduled job facepalm
Think it's back now...
I considered both of those and ended up using the External Secrets Operator + 1Password for my secrets. Maybe not the _best_ solution, but it saved a fair amount of effort on my part.
> Think it's back now...
It's back! Looking forward to reading!
My home lab has been rebuilt 3 times in the last 20 years and the continuous mention of energy usage within this post is a main driver and one of several primary objectives towards my effort. My 36U rack as configured now with Dell Servers having Intel Xeons E chips has served me well for all the tech stuff that interests me both in hobby and profession. My entire setup now with all devices running draws 200 watts continuously and my existing home energy renewables generation and battery storage system will greatly benefit from this home lab rebuild as I am targeting energy use and floor area recovery outright while learning many new things as well. This rebuild is also a lead into my fourth tech company I'm starting so my motivation is strongly tied to many factors not discussed.
Looking forward to the future posts, Stay Healthy!
Personally I have two Xeon rack servers running in a Proxmox cluster with a SBC Qdevice. It has more than enough memory and compute for my needs and it also serves as my virtualized router and NAS. The whole setup only takes up 4U of space (servers + switch + modem/qdevice) with a single UPS on the floor, and idle power is around 150W.
I focused on energy consumption, because of cost and - gasp - wanting to be mindful about it given the current predicament.
Anything that needs to be on 24/7 is on a Pi, and anything that consumes more power is turned off by default (remote poweron possible).
For me at home there is zero need for redundancy and I use a cluster of four tini-mini-micro 1L PCs for my lab work. There are also turned off by default and are also low-power.
I also got some cloud credits from my employer, but a bit paranoïd about putting my data there (although most of it isn't sensitive).
https://louwrentius.com/the-raspberry-pi-5-is-no-match-for-a...
That's the setup I've been using on 3 x rPi since 2021 and I'm super happy with it as I can host all my own personal projects, some OSS ones (changedetection, n8n, etc), even longhorn for distributed storage and still have capacity left -- and this with just microSDs (to be fair, A1s, but still).
It’s so much fun and helps me to own my data.
I'm sure nobody needs this much compute for personal use (24/7), so roadster in a garage is a good analogy.
I run 4 thin clients and only one of them is a worker. The rest are untainted control plane nodes in HA.
Confused as to why 10x nodes but running single control node and no HA?
Control nodes can be pretty light - even down to raspberry 4s