Comment by sliken | Hacker News Reader

Comment by sliken | Hacker News Reader

You sound like you are on a good trajectory. I'd start reading up on and implementing a firewall. Start with host based, then on a router/vlan firewall. Try out wireshark and view things like a simple wget, file download, and similar. If something weird is going on record with tcpdump (see wireshark docs for flags) and you can analyze afterwards with wireshark.

I'd recommend an IPv6 firewall if you can get IPv6, that exposes much complexity (I have 2^68 IPs on a normal consumer/home ISP connection). This will allow much of the complexity of a larger IPv4 LAN.

You sound pretty ambitious, just keep in mind that everything you mention is going to create state that you are responsible for. So implement backups from day 1. Last thing anyone wants is to lose all their photos, git repos, password database, etc.

Make sure your backups are in at least two places that can't be taken out by a single theft, flood, house burning down, company going out of business, etc.

Backups aren't backups until you verify them, do so regularly, maybe the 1st of the month or something. Verify files are exactly as backed up with sha256 or similar.

Specific recommendations: ZFS for any filesystem with 2 or more disks. Digikam for photo org and tagging, in a standards compliant way. Piwigo for self hosted photos ... that can use standard tags for organization.

SUPER helpful. Very grateful!

Generally new admins tend towards creating physical or virtual machines, logging in as root, editing files, starting services, and backing them up.

Which is fine, a great way to learn, a less great way to be in production. Upgrades can be painful, it's easy to break things is non-obvious ways, and there's no easy "undo".

Generally I'd recommend either using containers and maintain your build scripts in git, or use some configuration management software like ansible or puppet (there are many others as well) so you can install a new OS (virtually or physically), then have ansible/puppet/whatever add config files, packages, services etc to get things working. Of course then you keep ansible/puppet/whatever files in git AND back them up.

That way if there's some major upgrade of OS or software you can spin up a test env, test it, and then decide to promote/replace an existing machine ... or just ditch it.

Really good advice. I still have flashes of “Ohhhh righhhhhht it’s a VIRTUAL machine---I can...” make a snapshot, clone it and test new stuff, etc. etc. etc. I’ll be happy when it fully sinks in.

Generally I try to avoid VPSs. They are long lived, it's easy to accumulate undocumented state, and have substantial memory, disk I/O, and network I/O overhead. You may end up with a years old VPS with many changes over the years, and ugly things accumulate. Maybe you built a package from source, but now it's got a security vulnerability, but your forgot it's there. So apt upgrade doesn't help. Not to mention any security fix assumes you aren't already compromised. A rebuilt container (I suggest proxmox or docker) rebuilt will remove any previous existing compromise. Sure you can clone an old VPS... but you still don't know exactly what changes have been made, and if it was compromised the clone is as well.

Much better to have docker (or similar) build script, a ansible play, or a puppet manifest. They will clearly list needed packages, config files, services, at least in puppet's case you'll have the "trifecta". The trifecta is a mapping between a service, the required packages, and the config files. So Puppet can notice a config file change, restart a service, and if a package is missing install it. So puppet is to a degree self healing, even when an admin edits a file on the guest/container/VPS instead of in puppet and it's fixed on the next puppet run.

That way years later it's not big deal to upgrade the OS, or replace one piece of the stack with another, then apply ALL the changes you've done so far, and test the result.

I can tell you from experience it's a nightmare to inherit somebody else's VPS that has 3 python installs, tons of services, and requires manual intervention on boot to start/stop services. Would be MUCH better if it was just managed by config management and everything except the base OS is controlled by ansible/puppet.

Often custom VPSs are called pets or snowflakes (with a negative connotation), much better to make them easy to replace as needed, like cows (with a positive connotation) as the saying goes.

Wow, thanks. This is a good and useful perspective. I shall endeavor to make my VPS a source of nutrition rather than a source of affection. :-)

Can't reply to your reply, it's too deeply nested.

Even better than a VPS with multiple services is (IMO) a container per service. So you get a nice clear single function container, but don't have to pay the memory and performance overhead of having a VPS per service.

Containers have zero memory/performance overhead (it's based on cgroups, which normal linux processes have anyways), minimal disk overhead (not need for an OS install for each one), and saves a ton of ram (each VPS requires at least a linux kernel).

Reply on news.ycombinator.com