We're migrating many of our servers from Linux to FreeBSD
it-notes.dragas.net
it-notes.dragas.net
* Lack of systemd. Managing services through shell scripts is outdated to me. It feels very hacky, there is no way to specify dependencies, and auto-restarts in case of crashes. Many FreeBSD devs praise launchd, well... systemd is a clone of launchd.
* FreeBSD jail are sub-optimal compared to systemd-nspawn. There are tons of tools to create freebsd jails (manually, ezjail, bastillebsd, etc…) half of them are deprecated. At the end all of your jails end up on the same loopback interface, making it hard to firewall. I couldn't find a way to have one network interface per jail. With Linux, debootstrap + machinectl and you're good to go.
* Lack of security modules (such as SELinux) -- Edit: I should have written "Lack of good security module"
* nftables is way easier to grasp than pf, and as fast as pf, and has atomic reloads.
Maybe I am not tuning Linux appropriately, but I have been in situations where a Linux system is overwhelmed / overloaded and I am unable to ssh to it. I have never had that experience with FreeBSD -- somehow ssh is always quick and responsive even if the OS is out of memory.
Most of the systems that I deal with are Linux, but I still have a few FreeBSD systems around and they are extraordinarily stable.
I think I’ve grown to appreciate it, but not love it. It feels like both systems are at opposite ends of a pendulum.
The RC/unit system was very approachable for me. You could explain it easily, and you could get inside of it and mess around very easily. That affordance/discoverability was awesome.
With systemd, simple things are nicely templated. I can find an example and tweak it to achieve what I want. Complicated things get complicated real fast.
I love not having to track down pid files. I love not having to check if the pid file contents match a valid instance of the expected binary.
The RC system was simple and also full of exceptions and corner cases.
This is important, because this pid file is used to determine which process receives the kill signal. If you get it wrong, and have the right permissions, you can accidentally kill something that you did not intend to kill.
This is further complicated if you want to run multiple instances of Tomcat because now you need to have a unique path for this pid file per tomcat instance.
If the thing that you're trying to run doesn't fork, you then have to execute it in the background and then store the result of $! somewhere so that you know how to kill the process later on.
It's all very error prone and the process for each daemon is often different.
Okay so you have a PID file somewhere /var/myservice/myservice.pid. The contents of that file is a number which is supposed to correspond to a process you can find in proc.
But PIDs are recycled or more likely your PID files didn't get cleaned up on reboot. So you look at your file and it says 2567, you look up 2567 and see a running process! Done right? Well it just so happenes that a random other process was assigned that PID and your service isn't actually running.
pidfd's are the real real solution to this but the short and long tail of software uses pidfiles.
The *BSD rc systems are robust and rely on a framework that abstracts stuff out, you're never going to be digging in there for a pid file's location.
A lot of the staying power of Unix is that it's based on a very small set of simple, yet powerful, concepts. RC is one of them.
It's true that systemd is nowhere near as easy to hack/get into but at the same time I find that the largely consistent definition language/CLI give me a lot less reason to want to do so.
When the need does arise, there's usually a line or two I can add to my service definition to get it to do what I want.
I couldn't say the same about init scripts.
Systemd allowed to solve them in a reliable way at the distribution level with no need for the system administrator to do anything beyond providing the exports and fstab entries.
* set disk i/o schedulers to 'bfd' for spinning drives and 'deadline' for solid state, and 'none' for nvme, by creating a file in /etc/udev/rules.d . kernel must have deadline and bfd schedulers compiled in.
* turned on SCSI block multiqueue in kernel config. requires kernel command line option scsi_mod.use_blk_mq=1 to actually enable it. this helped, but did not completely cure the disk i/o problem.
* patched kernel source file ./block/blk-mq-sched.c to hard limit number of queued block device requests to 2, instead of default which is like 32. this absolutely cured the problem. no more disk i/o dragging the system down. doesn't seem to have a major effect on throughput.
* kernel is configured for full preemption, with 1000hz timer frequency.
* for architectures which will boot using the muqss cpu scheduler patch, i enable that with a 100hz timer freq instead.
* overcommit is disabled, as well as swap, and i use earlyoom to ensure process destruction proceeds in a controlled manner in event of memory exhaustion.
That's the bulk of it. No real magic involved; just un-fuck-ifying the default kernel config, which is garbage even for server use IMO.
(This is on a 4.x kernel btw, and I have no plans to downgrade to the 5.x series.)
The fact that these some or all of these tweaks aren't done by default would seem, along with other evidence, to support my belief that Linux is actively being sabotaged by people who don't want it to succeed.
Imagine running a restaurant, normally you can take 32 orders and have the customer sit and wait. One day you get new chefs that can make food 100x faster, but now you can only take one or two orders before the chefs have the food ready and you have to give it to the customer that ordered it. So despite the chefs being 100x faster it now takes much longer to place an order, and the waiting line can grow long with impatient customers.
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1208993
Ubuntu by default sets it to 2GB even if you have something like 16GB RAM. Swap file can then fill up pretty quickly.
Anything else, 2GB or less or not bother with a swap disc and just use a swap file with a cap at say 2x RAM. Also depends on how much RAM and disc is available.
My biggest annoyance, apart from the obvious lack of systemd, was a social one: Any time I had to look up how to accomplish some non-trivial task I would inevitably find a thread on the FreeBSD forums by someone else who has had exactly the same problem in the past, together with a response by some dev along the lines of "why would anyone ever need to do that?".
This is the principal reason why I've leaned toward Debian stable (which also aims at being a stable UNIX like FreeBSD) for decades, even though Debian has also been infected with systemd and has made other questionable decisions. Alternatively, I've also had good luck with Void, Alpine, and Artix. (I've had difficulties with electron apps on Void desktop, but it runs flawlessly on servers.)
Ansible, SaltStack, all the devops automation stuff is available and can certainly handle "freebsd-update fetch; freebsd-update install; reboot".
either that or "don't do an upgrade, build new ones and replace the old ones instead"
If by any chance it was related to sound, today is your lucky day as I just solved that this morning before work.
That was your mistake. With BSDs, you use the man pages ;-)
I don't think this has anything to do with graduating from berkely, I'm a hobo with a keyboard.
You may want to look at vnet, which gives jails their own networking stack; then you can give interfaces to the jail. If you use ipfw instead of pf, jail id/name is a matchable attribute on firewall rules; although it's not perfect, IIRC I couldn't get incoming SYNs to match by jail id, but you can match the rest of the packets for the connection. And that brings up the three firewalls of FreeBSD debate; maybe you had already picked pf because it met a need you couldn't (easily) meet with ipfw; you can run both simultaneously, but I wouldn't recommend it. Nobody seems to run ipf, though.
Edit: you may also just want to limit each jail to a specific IP address, and then it's easy to firewall.
No they are not. Systemd has a way bigger scope then launchd. It's like saying a truck is the same thing as a family car because they both solve the mobility problem.
> FreeBSD jail are sub-optimal compared to systemd-nspawn.
systemd-nspawn isn't a container. For example it doesn't manage resources such as CPU or IO. Again, the scope is way different and in this case there is a whole slew of things systemd-nspawn isn't going to manage for you.
BTW launchd doesn't have a feature like 'systemd-nspawn'.
> Lack of security modules (such as SELinux) -- Edit: I should have written "Lack of good security module"
And how is SELinux a good security module? SELinux with it's design fell in the same pitfall dozens of security systems did before it; ACL-Hell, Role-hell and now with SELinux we also have Label-hell.
It does[1]. At the end systemd-nspawn runs in a unit, which can be (like all systemd unit) resource-controlled.
> BTW launchd doesn't have a feature like 'systemd-nspawn'.
Neither does systemd. systemd-nspawn is just another binary, like git ant git-annex. The only difference from git and git-annex is that systemd and systemd-nspawn are maintained by the same team.
But like git and git-annex, most systemd installations don't have systemd-nspawn, but systemd-nspawn needs systemd.
> And how is SELinux a good security module? SELinux with it's design fell in the same pitfall dozens of security systems did before it; ACL-Hell, Role-hell and now with SELinux we also have Label-hell.
SELinux is generic enough to allow for sandboxing of services. But in its mainstream use, SELinux is well design that it allows for reuse of policies. Most people don't care about SELinux, they just run CentOS, install RPM from the repository, and everything works out of the box with heighten security. (= if there is a vulnerability in any of these packages, the radius-blast of the attack is limited thanks to SELinux's Mandatory-Access-Control)
[1] https://www.freedesktop.org/software/systemd/man/systemd.res...
That's not, strictly speaking, true. systemd-nspawn is able to do the cgroups-based containers fine without systemd as init, though you do lack some of the fancier resource control and networking features.
False.
> And how is SELinux a good security module?
It's good enough for most environments requiring RBAC. Additionally, it's complemented by the excellent sandboxing ability of SystemD unit files.
I hear it's taken over dns. Has it taken over sound too?
When will it be able to read mail?
systemd the family of tools has a DNS server, systemd-resolved. Like all other tools in the family, using the init system does not require using the other tools, and sometimes also vice versa.
In the broader "Poetteringware" family of tools, sound is handled by pulseaudio, though the server side of pulseaudio is in the process of being replaced by pipewire.
And btw is that why systemd using distros lock up for minutes on boot if the machine doesn't have internet connectivity?
This, this and this again !
After decades of cron, I discovered systemd timers via a passing comment I read here on HN.
My god is it amazing. No more hacky work-arounds in my scripts, systemd now takes care of all the magic such as random timings etc.
I'll never go back to cron.
http://fcron.free.fr/doc/en/fcrontab.5.html
1.0 released 21 years ago.
For example, you can schedule a timer to run 10 minutes after boot. Or a timer that actives 10 minutes after it has last finished running (note: not when it started last time but when it finished! So if the proc takes 10 hours, there is a 10 minute gap between runs. If it takes 10 minutes, there is still a 10 minute gap).
You can also schedule something 10 minutes after a user logs in (or 10 seconds later, etc.).
Additionally you get Accuracy and RandomizedDelay. The former lets you configure how accurate the timer needs to be, down to 1 sec or up to a day. So your unit now runs somewhere on the day it's supposed to run. And with the later you can ensure that there is no predictable runtime, this can be important for monitoring.
My biggest favorite is Persistent=. If true, systemd will check when the service last ran. And if it should have been scheduled atleast once since then, it'll activate. I use this for backing up my home PC. When I do a quick restart, no backups are done but when I shutdown for the night, first thing in the morning my PC has a backup done.
It will also run things that were scheduled to run when the server was turned off.
I'm not a fan of systemd personally but I do understand it has some good parts to it (such as the ones you've listed). That all said, you can still specify dependencies in FreeBSD with the existing init daemon. Albeit it's a little hacky compared with systemd (comments at the top of the script). But it does work.
> FreeBSD jail are sub-optimal compared to systemd-nspawn. There are tons of tools to create freebsd jails (manually, ezjail, bastillebsd, etc…) half of them are deprecated. At the end all of your jails end up on the same loopback interface, making it hard to firewall. I couldn't find a way to have one network interface per jail. With Linux, debootstrap + machinectl and you're good to go.
It's definitely possible to have one interface per jail, I've done exactly that. However back when I last did it (5+ years ago?) you needed to manually edit the networking config and jail config to do it. There might be a more "linuxy" way to do this with Jails now though but its definitely possible. eg https://etherealwake.com/2021/08/freebsd-jail-networking/
Security modules do exist, they're different from Linux. Are you sure you're not just expecting FreeBSD-as-Linux?
As for init... What can I say, I've never been anti-systemd, not even remotely, but rc.d is much nicer than sysvinit, and I find it much simpler to understand than systemd. In fact, I think rc.d is an example of how Linux could have alternatively migrated from sysvinit without pissing some people off.
You're right. My original message was wrong, I edited it, while keeping the original content. What I meant is "good security module".
SELinux on CentOS is "enabled by default and forget about it", unless you do something weird. MAC (= Mandatory Access Control) on FreeBSD requires much more configuration. They have some cool stuff like memory limits, but it's not as powerful as SELinux.
A fair bit is included in the base system already, see capsicum for example. Also, see HardenedBSD, which is arguably better than anything Linux has built-in.
No it isn't. There's a reason why government and military servers run hardened Linux with SELinux, and not any of the BSDs.
# PROVIDE: mumbled oldmumble
# REQUIRE: DAEMON cleanvar frotz
# BEFORE: LOGIN
# KEYWORD: nojail shutdown
> The `REQUIRE' keyword is misleading: It does not describe which daemons have to be running before a script will be started.
> It describes which scripts must be placed before it in the dependency ordering. For example, if your script has a `REQUIRE' on `sshd', it means the script must be placed after the `sshd' script in the dependency ordering, not necessarily that it requires sshd to be started or enabled.
I mean the shell is still used everywhere, like e.g. to configure and control systemd.
Still I would say a lot of core shell tools are indeed outdated (for backwards compatibility).
Your systemd unit file is backed by pages and pages of docs that must be comprehended to understand and hack on. Unix developers have all they need from the script. Furthermore, its all in context of existing unix concepts and thus your unix experiance is paying dividends.
As for 500 lines - take a look at proper rc scripts, eg the ones in FreeBSD. They are mostly declarative; it’s nothing like Linux’ sysv scripts, which were in some ways already obsolete when first introduced (runlevels? In ‘90s, seriously?)
Why is unlimited customization a good thing in the context of a system init?
This is the exact opposite of a good thing.
sort of. Systemd took a lot of lessons from launchd but also from sysv and upstart. For anyone who hasn't read Lennart Poettering's "Rethinking PID 1" post[1] I highly recommend it. You'll understand the history, but most importantly you'll understand systemd a ton better.
FreeBSD jails has quite a bit of history (developed in 1998) and made public in the year 2000. FreeBSD has had VNET for use with jails allowing you to add epair interfaces connected to a bridge, or add a physical interface or VLAN to a jail. This feature has been in FreeBSD since 8.0 (2009) and enabled by default since 2018. It also allows you to run PF in each jail.
Sadly many people a bitten by outdated forum and blog posts when it comes to jails.
I agree that firewalling jails with loopbacks is a pain, but most people don't do it that way anymore.
Also PF has always had atomic reloads.
Again, mature, simple and secure, with very few surprises lurking. There's likely fewer lines of code to support all jail functionality than in systemd.
I suppose you didn't use vnet? It's a vastly better jail networking experience. You can pretend jails are separate machines, connected via ethernet. I don't think anyone who knows about vnet chooses not to use it!
> I couldn't find a way to have one network interface per jail.
I think vnet is what you want.
> nftables is way easier to grasp than pf, and as fast as pf, and has atomic reloads.
pfctl allows you to do atomic reloads. `pfctl -f`
I'm not sure initv "worked okay".
I think it was a ZFS dev who complained he had to update 150 files to port ZFS to systemd. Simples?
And you have to love all them binary log files, with their dynamic and spontaneus api. Yes, everything is always new and exciting with systemD.
I use devuan. A debian fork with a choice of init systems (well, everything but systemD. ;) It took the dev/devs 2 years just to swap out/revert the init system.
Also, just learned devuan's website is JS and cookie free. https://www.devuan.org/
Edit: I will concede that systemD is great for a lot of people. I honestly wish them success. The work that goes in to building, maintaining, and using it, is substantial. It must be a boon for the economy and job creation.
Seriously, though, read the article. The 1.2M lines includes removed lines from refactoring. So, it is not completely made up, as implied in that comment. https://www.phoronix.com/scan.php?page=news_item&px=systemd-...
https://twitter.com/pid_eins/status/1214268577509003266
https://www.phoronix.com/misc/systemd-eoy2019/files.html
Including the number of lines in static tables and documentation hardly seems like a meaningful comparison.
I suppose that the end goal of the systemd project is an ability to deploy a production Linux system with just systemd and busybox, and run all software from containers.
Not that it's a bad thing to strive for. But it's not going to be Unix as we know it.
To be more specific is it a replacement of more and more Linux userland that nobody asked. I would be ok if systemd would be a process management system that replaces init with a standardised way of managing services (it would be amazing if it was written a safe language, if it was respecting configuration files, if it did not take over managing system limits, etc. etc.).
Unfortunately it is trying to do too much.
(I'm still able to run Void Linux as my primary desktop without systemd.)
Right, that's Plan9.
(Yes, I do regret that Linux was a hyper-succesful project that replicated Unix, not Plan9. But Plan9 is more complex, maybe it was just infeasible.)
Serverless is killing containers. Its killing the need to care about if its freebsd or linux, does it run my api fast enough?
> The whole system is managed by the same team
Mostly philosophical.
> FreeBSD development is less driven by commercial interests.
Mostly philosophical.
> Linux has Docker but FreeBSD has jails!
IMO, this comparison is a mistake. In the Linux world, systemd's nspawn is very similar to Jails. It's a glorified chroot, with security and resource management. All the systemd tools work seemlessly with nspawn machines (e.g. `systemctl status`). Containers à la Docker are a different thing.
BTW, I thought the last sentence about security issues with Docker images was strange. If you care about unmaintained images, build them yourself. On the other side, the FreeBSD official documentation about Jails has a big warning that starts with "Important: the official Jails are a powerful tool, but they are not a security panacea."
> Linux has no official support for zfs and such
Fair point, though I've heard about production systems with zfs on Linux.
> The FreeBSD boot procedure is better than grub.
YMMV
> FreeBSD's network is more performant.
Is there some conclusive recent benchmark about this. The post uses a 2014 post about ipv6 at Facebook, which I think is far from definitive today. Especially more since it "forgot" to mention that Facebook intended to enhance the "Linux kernel network stack to rival or exceed that of FreeBSD." Did they succeed over these 8 years ?
> Straightforward system performance analysis
The point is not about the quality of the tools, but the way each distribution packages them. Seems very very low impact to me.
> FreeBSD's Bhyve against Linux's KVM
The author reluctantly admits that KVM is more mature.
I like your point about Docker vs Jails, I haven't seen it discussed like that before. I keep hearing Jails are more secure than anything else, I'll have to read more into it.
As far as the networking goes, I haven't seen any recent benchmarks to substantiate those claims either. However, considering Netflix uses FreeBSD on their edge nodes and has put a lot of work into the upstream to improve networking (among other things), it wouldn't surprise me if it's technically superior to the Linux stack. Clearly though Linux's networking isn't an issue for most organizations.
And regarding ZFS, ZFS on Linux and FreeBSD's ZFS implementation are now one and the same. It would be nice to see some of the big distributions(or even the Linux kernel) integrate it more directly. This is probably a solid point in favor of FreeBSD, but it's not like it doesn't work in Linux. I'm not a systems guy, so I'm probably out of the loop on this, but Proxmox is the only distribution I've seen with ZFS as a filesystem out of the box, but I don't know how much production use Proxmox sees. I only run it on my home server.
All that to basically say, I like FreeBSD conceptually. I'm just still not convinced that it's doing enough things better to warrant using it over a common Linux distribution for general computing purposes.
Possible. But it's also possible that the founder of the team was a BSD person. You occasionally get cases where the preferences of one key person affects things in the long run. At this point I'm sure that they're baked in, because removing all that code would not be worth the effort too, so even if it was better when Netflix went online, that doesn't guarantee that it's still true today.
That being said, Linux is also used in a ton of places, including high network jobs. It would take a decent sized amount of evidence to convince me that all of those other places were wrong and basically only Netflix was right to use BSD in network heavy cases.
My suspicion is that either:
1. The difference is minimal to negligent, and not enough to justify mixed OS development
or
2. The difference is significant, but you have to be pushing your network much harder than most teams ever do for it to show up.
FreeBSD networking during WhatsApp and Netflix starting era was definitely better than Linux. Both in their specific usage pattern. So it wasn’t some ideological reason but actual technical one. But that was 10+ years ago.
If I remember correctly from reading Meta / Facebook successfully switch all of WhatsApp large bare metal 4U server to their standardise Linux blade by 2017. They also spent 3 years working on Erlang BEAM and Networking so it was working close enough.
Netflix continues to use FreeBSD and it is working well for them. Both the lead engineer comment and submit update on HN frequently. Do a search and it should be easy to find it. ( on my phone now so can’t quote links )
2022 now I think any general advantage of FreeBSD over Linux in networking performance would likely be minimal. But it also doesn’t make sense to switch to another for the sake of OS unification.
And say what you will, something like 20% of all internet traffic has a FreeBSD endpoint. And doing 400Gb/s of encrypted streaming from one box is quite an accomplishment.
I would argue the reason someone like Netflix or any of the other large orgs using FreeBSD come there is for the simplicity/cohesiveness. If you're looking to do something like in-kernel TLS or something, way easier on something smaller, documented, and with an OS devel team that will likely incorporate your work in future releases.
https://github.com/freebsd-net/netperf/blob/master/Documenta...
Don't assume that this means feature parity, not by a long shot. ZFS on FreeBSD affords you native NFSv4 ACLs (hugely important in corporate settings), tight integration with jails, much better delegation support, the FreeBSD boot loader is actually updated to understand all the ZFS features, and offers boot environments and checkpoint rollbacks right from the boot loader menu (and not having to shuffle around with a dedicated /boot partition like you do on Linux).
If none of those features appeal to you, ZFS on Linux will seem very much the same. Indeed, within the limitations of Linux itself, ZFS does very well at providing every feature that Linux lets it provide.
One of the things that annoys me to no end ( and I am not sure if this is something specific to Netscalers or FreeBSD networking appliances in general ) is that the damn VM's are the only thing I run that does not properly report memory and CPU usage to VMware. From the hypervisor perspective they are constantly running at 100% CPU (their support has told me that the system reserves the full amount of CPU given but is not actually running that high - and when you actually connect to the box it self reports the number correctly). Not a huge deal but it annoys me to have that little red "X" in the GUI next to the VM name all the time.
That being said I also have not seen any recent FreeBSD vs Linux benchmarks but I would imagine if a company as large as Netflix is already using it en masse then there would need to be not just parity but tangible benefits to swap it all out for some other linux distro. An org starting from scratch of course would be a different beast.
And if you never see FreeBSD again... So what? The stuff you are building on top is probably more relevant than questions like this.
> Many times you also need to think about what may not be the very best solution from a purely technical perspective but also what is the best from both a people and technical perspective.
This is huge when it comes to running the company's software imho. It's useless having the perfect solution if you can't get anyone to run it when "good enough" will have a plethora of folks ready and willing. Especially when it comes to managing the bus factor. When I asked the admin at my org about that, he explicitly said he didn't care about it (that's more an indictment of him than FreeBSD though).
Overall the human factor is something I've been trying to embrace more lately when it comes to work. For personal stuff though I'll be as esoteric as I like.
> the damn VM's are the only thing I run that does not properly report memory and CPU usage to VMware
Well that's just plain frustrating.
What's holding this back is legal issues, since ZFS's licenses is not compatible with the GPLv2.
I would not be surprised if there still were some cases where FreeBSD could do better. The network stack in particular has a vast amount of performance heuristics and tuning where you can just happen to hit the right or wrong side of some performance jump depending on your exact case, for example.
But to go from there to the network stack is more performant is puzzling. There are certainly cases where the Linux network stack is more performant than FreeBSD -- search for linux vs freebsd network performance and you'll find benchmark tests and complaints and anecdotes around the place including freebsd forums and mailing lists where FreeBSD is slower, and yet FreeBSD proponents would (rightly) say these are not proof Linux's network stack is faster and there might be all sorts of reasons for the differences. So you can't have it both ways.
Netflix uses FreeBSD for some things yes. Not sure if number of big internet corporations using an OS would be a favorable metric for FreeBSD either. Again you can't have it both ways. Netflix says FreeBSD is capable of performing well in that kind of environment, it does not say that it outperforms Linux in general or even in that environment. Any more than Google says the opposite.
I don't know about Facebook-driven efforts specifically offhand, but I recall attending a talk at LinuxCon 2015 _specifically_ about optimizing the network stack. Given by a Red Hat employee, IIRC.
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...
From a quick glance through I see a lot of work on eBPF stuff, memory, I/O subsystem, cgroups, and btrfs.
> Like all other systemd-nspawn features, this is not a security feature and provides protection against accidental destructive operations only
Doesn't seem very similar to jails to me.
But to imply that these are empirical differences between FreeBSD and Linux, well, that's nonsense.
I don't really disagree with any of the stated reasons, but I also didn't see a reason that would make me even consider making the move with our servers, or even bother with some small number of servers. At least for me, I'd need a bunch of REALLY GOOD reasons to consider a move like that. A huge cost savings AND some huge time savings in the future might do it.
But they left out one of the bigger reasons, IMHO. FreeBSD doesn't tend to churn user (admin) facing interfaces. This saves you time, because you still use ifconfig to configure interfaces, and you still use netstat to look at network statistics, etc; so you don't have to learn a new tool to do the same thing but differently every couple of years. Sure, there's three firewalls, but they're the same three firewalls since forever.
Personally I blame all of the major Unix-derived operating systems (Linux, macOS, BSDs), as none of them show any interest in standardizing any APIs or commands invented this millennium. The subset that’s common to all of them is frozen in time, and is slowly being replaced by new bits that aren’t. From epoll/kqueue to eBPF, containers/jails to seccomp/pledge, DBus/XPC to init systems… from low-level network configuration (ifconfig, ip) to high-level network configuration (whatever that is this month).
On FreeBSD, the last commit on it was last week.
They're not the same tool, and FreeBSD didn't abandon their core tools, because they're part of the base system.
https://sourceforge.net/p/net-tools/code/ci/4030929bb6f3ee6f...
It's still used in some user spaces, and the Linux kernel supports the system calls for net-tools still today, because of the Torvalds Prime Directive, "though shalt not break userspace".
Likewise, some people and many businesses see the immediate now and aren't always the best at planning for long term, and/or are overly optimistic that their pain points will eventually be fixed.
But that's the thing. The pain points mentioned in this article aren't really that strong to need to ditch the OS and move to a new one. These kinds of decisions have huge tradeoffs.
One could argue, in fact, that moving to a non-traditional OS will make it much harder to hire experts or hand off the system to another team in the future.
After CentOS Stable was cancelled, I migrated a number of our platforms to FreeBSD because it met our needs and I enjoyed working on it. No surprises, nothing breaking, and most importantly, no drama.
Nothing there seems to deliver explicit customer value when switching to FreeBSD.
How will switching help you deliver your service? Or is it just a "nice to have thing"?
If your backend OS starts to run software more efficiently, cost less to host, has less maintenance downtime, has fewer security incidents, has fewer crashes, etc, having changed to it has produced customer value.
Is the service working, is it not? (are there show stopping OS issues?)
Are the performance characteristics we need there, or not?
Is the service secure or not?
There's a whole lot of things right through that entire space where the choice of operating system can make quite a fundamental difference.
As a general statement, I wouldn't want to make this kind of a migration without something bordering on a killer feature that wasn't possible otherwise, or some fundamental driver problems (I've replaced Linux with FreeBSD on border devices in the past, due to particular issues with Linux in ways I couldn't afford to have keep biting me)
> 70 binaries just for initialising and logging
It’s just not true. Those 70 binaries provide much more functionality than an init system, they can cover a significant portion of system management, including a local DNS resolver, network configuration or system time management. You can dislike the fact everything is so tightly integrated (which feels ironic given that the post goes on to praise a user space from one team), but let’s at least be correct about this.
Do. Not. Want.
So, I have this pile of 70 binaries that are inexplicably tied to my init system, and (in my, informed enough for me, opinion) they're all garbage. How do I remove them without breaking init?
This month's fresh hell: I have a problem where the systemd replacement for xscreensaver (logind, maybe? Good luck finding the culprit, let alone the manual!) won't accept my password unless I exit the current X session with "switch user" then restore the session using the normal login screen.
There's a whole section on JWZ's xscreensaver page (from over a decade ago) explaining how to avoid this class of bug, but what does he know?!?
That reminds me; I wonder if a *BSD is a good enough daily driver for the pine book pro yet (it's probably easier to port their kernel than to fix Linux userspace, after all...)
Depending on distro, all can be substituted for alternatives (assuming they're used at all, I've seen a number of distros package the kitchen sink "just in case")
Nothing about using systemd as pid1 required using any of their other developed tools, even logind. Find a distro that doesn't use them, or roll one yourself.
It's a mistake to see "Systemd" as a single application - it's a collection of developed-together (so tend to work slightly nicer together) userspace tools - like FreeBSD without the libc and kernel, or GNU coreutils or similar. People don't seem to complain too much about those "Bundling Everything Together".
Docker, podman, lxc, lxd, etc are userland components. Linux has cgroups and namespaces.
FreeBSD jails are a bit more complicated because FreeBSD isn't distributed the way Linux is. Linux is distributed as just the kernel, whereas FreeBSD is a base OS. This probably could've been phrased better as, "Linux has no interest in userland and I want some userland consistency". That's fair, Linux was built around the idea that operating system diversity was a good thing long term, FreeBSD was more interested in consistency. I'm reading between the lines, a bit, here because of the critique of SystemD (note: not all linuxes use SystemD)
Personally speaking, I like both Linuxes and FreeBSD but I don't think debating the two is important. Rather, I'd encourage turning your attention to the fact that every other component on a system runs an OS-like interface that we don't make open OS's or "firmware" for.
This is really telling for the level of analysis done: systemd has been the target from a small number of vocal complainers but most working sysadmins only notice it in that they routinely deal with tasks which are now a couple of systemd stanzas instead of having to cobble together some combination of shell scripts and third-party utilities. Confusing noise with numbers is a dangerous mistake here because almost nobody sits around randomly saying “this works well”.
To sum up: healthy FLOSS competition is welcome and needed.
This may not be the best place, but I have to. I'd like to tell you that, although we've had some disagreements on HN, I carry no bad feelings and, for as much as possible, have extreme respect for you and your opinions.
I'm glad our previous disagreements don't prevent us from posting when we do agree.
So it ok to agree to disagree. :)
There's too many decades of cruft and backwards compatibility built up. Afaict, most interesting new OS's being built right now are similar to UNIX, but very explicitly not.
In terms of innovations, I'd bet MacOS has evolved too. Although "logical partitions"-like solutions were already known for some time, Linux made it widespread through containers; io_uring allows high throughput syscall-less zero-copy data transfer and futex2 allows to implement NT synchronization semantics that are very common in game development. All that ignoring just how much the desktop changed!
The UNIX children are definitely not sitting still.
Then again they would probably charge 10x as much for a 2U rackmount server where the main difference was a nice stainless steel front fascia...
Where is this coming from exactly? The linked article about Facebook is 7 years old. The following benchmark shows the exact opposite: Linux's network stack has long surpassed FreeBSD's. And I would expect nothing else given the amount of work that has gone into Linux compared to FreeBSD.
https://matteocroce.medium.com/linux-and-freebsd-networking-...
However, I'm up to serving 709Gb/s of TLS encrypted traffic to from a single host to real Netflix customers with FreeBSD.
Basically just wondering if both of you are correct and the default implementation of FreeBSD is in fact losing to other flavors of Linux but the networking customized OS' (not even just thinking of Netflix at this point) are still superior.
Most of the changes that I have for the 700g number are changes to implement Disk centric NUMA siloing, which I would never upstream at this point because they are a pile of hacks. They are needed in order to change the NUMA node where memory is DMAed, so as to better utilize the xGMI links between AMD CPUs.
Or is that Single CPU? Because I thought you try to avoid the complexity of NUMA. Which I remember it was causing issues with first gen Zen.
Its mostly an experimental platform, never intended for production, to shake out issues that we'll see when next generation platforms show up with DDR5 and PCIe Gen5. With a current single-socket setup we'd be limited by memory bandwidth around 500Gb/s or so, but 800Gb/s should hopefully be within reach from a single socket with DDR5 based servers. Especially since the limiting factor now is the traffic on the xGMI links between the sockets is unbalanced.
https://netflixtechblog.com/serving-100-gbps-from-an-open-co...
I'm glad there are some people who use and prefer FreeBSD and other init system now, because diversity in digital ecosystems is benefits the whole just as diversity in natural ecosystems do.
The shot taking at systemd here was disingenuous though. The author complained about the number of different systemd binaries and the lines of source code, but all these tools provide a highly consistent "system layer" with standardized conventions and high quality documentation-- it's essentially the same argument made to support FreeBSD as a large body of kernel and userspace code that's maintained in harmony.
Nothing wrong with the comparison probably, but I got the impression the author just really wanted to do the migration and found some reasons to do so, without actually needing it. Nothing wrong with that as well. It's just the expectations set by the title that are off
I think many of us (including me) have a tendency to try to quickly generalise our experiences, even when it's not appropriate - and when we go onto explain things to others without the original context it can sound too abstract or come across as evangelical. Either way, it loses meaning without real examples.
If FreeBSD jails and Solaris zones were equivalent to Linux containers, we'd have seen them take over the backend already. We haven't. They're really useful, they provided a degree of safety and peace of mind for multi-tenancy but they're not granular enough for what's done with $CONTAINER_RUNTIME these days.
Jérôme Petazzoni has an old talk where he touches upon container primitives and compared them to jails : https://www.youtube.com/watch?v=sK5i-N34im8
Are there any FreeBSD-centric answers to Ceph or Gluster or Lustre or BeeGFS?
It's not BSD centric but it works.
https://docs.freebsd.org/en/books/handbook/disks/#disks-hast
Might be a future alternative in the space.
https://gitweb.dragonflybsd.org/dragonfly.git/blob_plain/HEA... was last updated in 2018, and at the time much of the clustering logic is described as "under development" or "not specced."
I've been using Linux since I was a teenager, so it's not like I am a stranger to fixing driver issues, but it seemed like no amount of Googling was good enough for me fix these problems (googling is much harder when you don't have functioning wifi). As a result I've always just stuck with Linux (or macOS semi-recently, which I suppose is kind of BSD?).
My take: I love FreeBSD as a server os. It's really, really well designed. Spend a bit of time with the handbook and after a while it's really simple to hack on. I really like the separation of the base OS, I like the init, I like jails, documentation puts every Linux distribution to SHAME.
But laptop?... Unless you scour for exactly the parts that will work (esp for wifi), you're going to have a bad time, even on old equipment. At the same time as I rebooted my server, I had an old ThinkPad I decided I wanted to FreeBSD for the hell of it. I gave up on it. It may have been possible but it was just too much work. In this day and age when almost any Linux desktop distro boots without a hitch on hardware that's not completely brand spanking new, it was just not worth it
Usually when this happens in Linux, a few hours of Googling, swearing, and retrying is enough to fix my problems, and I'm sure that with enough time that approach would have worked in FreeBSD as well, but I always grew impatient.
> almost any Linux desktop distro boots without a hitch on hardware that's not completely brand spanking new
Can't speak for anyone else, but even for brand spanking new hardware, as long as I've stuck with AMD drivers, nowadays Linux "Just Works" when I boot it. Obviously YMMV between systems, but I feel like Linux has finally become competitive with Windows and macOS from a usability standpoint.
Honestly I just wish there was a way to run ZFS as a core partition on macOS. APFS is pretty good, but a full on ZFS thing is really cool, and makes FreeBSD so much more appealing as a result.
I set up Hyper-V for the first time last summer, and I seem to recall it just working.
I've read hundreds of post surrounding this issue. I've used laptops for decades but I don't think I have ever used this feature. It doesn't take even a second to hit a couple keys and sleep a laptop. The only feature lower on my priority list would be syncing the RGB keyboard to soundcloud. But that's just me. Evidently the lid-close-sleep thing is of vital import to millions of laptop users. It's one of those things where I just shake my head in bewilderment.
> Evidently the lid-close-sleep thing is of vital import to millions of laptop users.
It's one of many little things that takes a laptop from feeling like some weird barely-functioning misfit of a toy to an actual portable tool that you don't have to worry about or baby.
hw.acpi.lid_switch_state=“S3”
Some of the distros are very good, but some of us who have work to do cringe at the thought of bringing up newer versions of an OS just to check all the things that've broken and changed needlessly.
There were other choices that Linux could have taken rather than sysvinit vs init.d and some distros through the years have taken different approaches. FreeBSD's rc.d is not sysvinit either.
I don't think that is true. If you look at those other distros you'll mostly find that it either didn't pan out, or it's quickly approaching the same design and architecture of systemd, where distros start using declarative prgramming to integrate tooling around common workflows. This is inevitable when you consider the constraints on the system as a whole. Other related service management tools like Docker are under the same constraints and you'll notice those have a similar architecture too.
I've run a lot of Linux machines on a lot of hardware for a lot of years and have no idea what could have been going on that would cause a 5+ minute boot, that was also unnecessary enough that a different BIOS could get it down to 30s. Back in the bad old days of circa ~2000 when I ran Linux on a variety of garage-sale potatoes, I don't think any took 5 minutes to boot.
Their first reason is really saying a lot but with few words. For one, there’s no systemd. The init system is maintained alongside the entire rest of the system which adds a lot of consistency. The documentation for FreeBSD is also almost always accurate and standard. Etc etc
I think you also largely don’t need a docker or etc in it since jails have been native to the OS for decades. I’d want to do some cross comparison first though before committing to that statement.
Shouldn’t be lost that the licensing is also much friendlier to business uses. There’s afaik no equivalent to rhel, for that matter. This goes both ways though as how would you hire a FreeBSD admin based on their resume without a rhce-like FreeBSD certification program?
Edit-I’ll posit that since FreeBSD is smaller an entity wishing to add features to the OS might face either less backlash or at least enjoy more visibility from the top developers of the OS. Linus, for instance, just has a larger list of entities vying for his attention on issues and commits.
On that playing field, the integrated nature of FreeBSD is nice but it’s an asterisk on top of the kernel rather than anything approaching what makes up a the system part of an Operating System. Almost everything else comes from a third party (and I’m fine with that.)
I haven’t used FreeBSD as a daily OS for over a decade though. What’s the new coolness?
I think the point still stands though. There has been lots of stabilization work done to BTRFS, and anyone using production-recommended features should consider the filesystem stable.
I slap Linux on my servers and desktops and I deploy FreeBSD via pfSense on firewalls.
Sometimes I do experiments and try out BSD on the desktop which hasn't worked out yet for me but I live in hope because I adore *BSD as much as I do Linux.
If BSD is the way to get your servers to do what you want then lovely. Do it and remember you have choice.
When I compare to ubuntu which is the OS I use for linux mostly now: * kvm is superior to bhyve in every way * automating security updates via apt are better than a combination of freebsdupdate/pkg updates. Plus the deb packages are made by Ubuntu and just work. ports/pkgs are third party on freebsd * rebootless kernel updates exist for ubuntu * It is easier to find people familiar with linux right away
Really though the learning curve of freebsd <-> linux is not high.
Attempts were made to fix it, but they eventually decided to ship 13.0 without wiregaurd.
It was very policitcal because the company that sponsored the development had already promised the feature to customers.
Even if an OS is 'better', a VM with a short lifetime will generally be 'good enough' very quickly. If you add a very large ecosystem and lots of support (both open source and community as well as commercial support) and existing knowledge, FreeBSD doesn't immediately come to mind as a great option.
If I were to go for an 'appliance' style system, that's where I would likely consider FreeBSD at some point, especially with ZFS snapshots and (for me) the reliably and fast BTX loader. Pumping out BSD images isn't hard (great distro tools!) and complete system updates (due to the mentioned "one team does the whole release") are a breeze as well. This is of course something we can do with systemd and things like debootstrap too, but from a OS-image-as-deployable perspective this will do just fine.
This latest version has something called "journaled soft updates" and it's a metadata-journaled system, i.e. the actual data is passed through, and it's non-CoW.
If your disk is slow or dieing, you might blame UFS, but it's not really UFS.
I've had some vague issues with the I/O scheduler, which isn't really UFS, but at the same time, UFS may be the only real client of the I/O scheduler, I think ZFS does it's own thing, anyway the systems were UFS only. This is super vague, and I don't have more details, but I just want to put it out there. For one class of machines that had a lot of disks (about 12 ssds), did a pretty even mix of reads and writes, evenly spread across the disks, upgrading from FreeBSD X to X + 1 wasn't possible because there was a large performance reversion. I think this was 10 -> 11, but it's possible it was 11 -> 12. Because this came up while my work was in progress migrating to our acquirer's datacenter which included switching to their inhouse Linux distro, it made sense to just leave those hosts on the older OS, and not spend the time debugging this. We didn't have a way to test this without production load, but that had user impact, and it would take a while to show up. It's quite possible this was just a simple tuning error, or possibly a bug that has been fixed for some time; the symptoms were obvious though: processes waiting on io, but the disks had a lot of idle time.
If you have a lot of files in a given directory, that's kind of slow, and IIRC, if the directory every had a lot of files, the extra space won't get reclaimed until the directory is deleted, even if most of the files are unlinked. (This isn't uncommon for filesystems, some filesystems handle it better than others, there are application level strategies to deal with hashing/deeper directory trees)
If the filesystem ever gets too full, the strategy to search for free space changes to one that's less fast; it won't change back, so don't fill your disk too much. (This ends up being a good idea for SSD disk health too, and again isn't super unusual in filesystems, but some filesystems probably do better). tunefs(8) says:
> The file system's ability to avoid fragmentation will be reduced when the total free space, including the reserve, drops below 15%. As free space approaches zero, throughput can degrade by up to a factor of three over the performance obtained at a 10% threshold.
UFS has snapshots, which is great, but everyonce in a while, you end up with a snapshot you forgot about, and it can really eat disk space and you may miss it. Not really a performance issue, but can lead to overfilling your drive.
Of course, there's the obvious that UFS has no support for checksumming, but that's not performamce. Soft updates do allow for some amount of consistency in meta data, and background fsck is nice (but could tank performance, I suppose).
I haven't made much use of them but the mirrors or raidzs seemed to perform more or less inline with expectations (consumer hardware may not have the PCIE lanes really available to run multiple fast NVME devices well).
Compare with XFS over a mdadm say in raid10 3 legs f3, then cry.
> consumer hardware may not have the PCIE lanes really available to run multiple fast NVME devices well
Trust me, I have all the lanes I need, even if I would always wish I had more :)
for instance, what is your ARC configuration in this case? It can have a massive impact of performance.
getting ZFS to perform well takes a bit of work, but in my opinion performance is on par with most filesystems. (and it has a ton of additional features).
L2ARC can improves cached reads, but it's not magical, especially not for random reads... or writes. (and yes, I know about SLOG, but doing async is faster than improving sync)
And don't get me started on how ZFS is not using mirrors to improve read speed (unlike mdadm can do, cf the difference between o3 n3 f3) or how it can't take advantage of mixed arrays (ex: a fast NVME + a regular SSD or HDD to add redundancy: all the reads should go to the NVME! The writes should go async to the slow media!)
If you don't have a RAID of fast NVMe that are each given all the lanes they need, you may not see a difference.
But if you are running baremetal close to 100% of what your hardware allows, and the choice of everything you want to buy and deploy, you'll see these limits very soon.
In the end, I still chose ZFS most of the time, but there are some usecases where I think XFS over mdadm is still the best choices.
direct i/o?
> I sometimes experienced severe system slowdowns due to high I/O, even if the data to be processed was not read/write dependent. On FreeBSD this does not happen, and if something is blocking, it blocks THAT operation, not the rest of the system.
I’ve seen this for a long time in Windows, where any prolonged I/O brings the entire system down to its knees. But it also seems to affect macOS (which is based on FreeBSD) as a system, though it’s not as bad as on Windows. Has Windows improved on this over the years? I’m unable to tell.
That's somewhat overstated, so I wouldn't draw such conclusions. (https://wiki.freebsd.org/Myths )
Computers get faster, we throw more at them.
Physics is still the law of the land.
Truh truh truh trade offfss; in computer eng; truh truh truh trade offs sorry Bowie
Curious to know the reason.
Reason I want to know is when I tried running ubuntu off a spinning disk and wanted to move some games to the drive it would transfer about 1gb then lock up. Didn’t happen with an SSD.
That being said it also forced me to use FreeBSD a lot more than I ever would have otherwise and I have a lot of respect for the OS itself. I would not use it everywhere but it has amazing latency which makes it obviously great for networking.
Related: https://www.youtube.com/watch?v=o_AIw9bGogo -- The tragedy of systemd.
Where Benno Rice (FreeBSD Committer / FreeBSD Core member) explains the value of something like systemd.
"FreeBSD's network stack is (still) superior to Linux's - and, often, so is its performance."
This is wrong, if it was the case most large compagnies would use BSD, atm they all use Linux, the only large compagny using BSD is Netflix because they added some tls offloading in kernel for their CDN which could have been done in Linux btw.
imo don't use tech that is not widely used, you're going to reinvent the wheele in a worse way because tool a.b.c is missing.
so kinda windows with wsl v2 is the way to go
https://www.cl.cam.ac.uk/teaching/1516/ConcDisSys/2015-Concu...
I run FRR Routing punching 400gbit+ @ 75% CPU load on FreeBSD 12.
On the same hardware, Debian fell over at 1.8gbit.
I have been using FreeBSD for networking for awhile now but not at those levels
You should know that due to license incompatibilities (CDDL and GPLv2), it's not really as smooth as the BSD integration (I wonder how Canonical avoids this issue). You know that OpenZFS's codebase is a monorepo (for the most part) but the in-kernel implementation is vastly different.
I use Zfs as my boot file system on FreeBSD and Arch Linux and both work great. Linux was a bit of a faff to set up though, FreeBSD was easy.
My question is how well moving their systems will shake out longer term.
I guess to each their own but I dislike docker. I think it’s bloated and over complicated.
There's nothing wrong with switching to another OS, but implying it's because the other OS is somehow empirically "better" is misguided.