Hacker News Reader

Comment by kwhitefoot | Hacker News Reader

So why do we have to wait so long for other versions to boot?

On many machines, the firmware is the largest contributor, with user-space services and their dependencies being the rest.

`systemd-analyze` (in particular, `systemd-analyze critical-chain` and `systemd-analyze plot > plot.svg`) are useful to diagnose. On my system you can see that my graphical session waits for systemd-user-session.service, which waits for network.target, which waits for NetworkManager taking its sweet time, which isn't started before network-pre which waits until nftables is done loading rules, etc.

Optimizing the service order and their dependencies does wonders for boot time.

> Optimizing the service order and their dependencies does wonders for boot time.

Why isn't this automatic? I'm pretty sure most of us have neither expertise not enthusiasm for that job.

It is automatic, but systemd obviously is constrained by the dependencies specified inside the unit files (that's kinda the point). The real question would be, why can't NetworkManager handle being started before everything is initialized. In turn the answer is probably "because it's hard"

I mean, it is, within the rule set specified by the config files. Distros just tend to explicitly default to the system being more fully up before presenting an interactive GUI because a lot of users' workflows expect that.

It is a matter of requirements. A login may require LDAP or Kerberos, which requires network connectivity. Pulling up network before the firewall is initialized could be dangerous. Maybe your desktop environment needs to play a jingle, so it by default waits for sound to init.

Service optimization is basically deciding what your system doesn't need in order to function for your usage, and moving things you don't need to a late initialization with no dependents. A distribution dedicated for a particular machine usage and desktop environment could do some tuning on your behalf, but it is not generic.

A lot of it is hardware startup and enumeration, and also software services (because they had to really strip it down to get there).

I'm more curious why my car, which is fairly recent, doesn't start in 0.3s.

There should be a key you can hold down which would mean "skip all enumeration and assume everything is exactly as it was the last time you booted" (with a brief warning to the user about the possible risks introduced by this)

If we're going that route, I'd like a bootloader-supplied kernel option that defaults to enabled (ideally with the GRUB/whatever menu having am option to disable it.

That's fair. I just wanted it to default to safe (as good user design dictates).

A typical car is broken down into many small, dedicated computers.

The powertrain and body control modules, assuredly running a RTOS, certainly do. The PCM for example controls when the spark plugs fire. If it did not boot instantly, the engine wouldn't run. If the BCM didn't boot, the doors wouldn't unlock nor would the key or PTS start the engine.

The user-facing infotainment crap is not necessary for the "car" part to function. You could physically remove it and the car would still be drivable.

If you turn off quiet boot, you'll see an impressive list of things getting started up, many of which the kernel isn't in full control of. DHCP (sometimes), gpus, NTP, disk decryption, TPM, monitor probing, USB hub enumeration, on and on.

But all the stuff that is actually necessary is the same on every boot so no enumeration and probing should be necessary.

Is it? My laptop is only shutoff when switching from a dock at home to a dock at work, changing the GPU and hubs available. The OS I just netbooted off my NAS has no idea what it's booting on. I can plug and unplug USB devices anytime, including while the system is off. Maybe I swapped USB headsets, maybe I plugged in a fingerprint or smartcard reader, which needs to be initialized before the login manager starts. Maybe my time server is no longer reachable, so NTP needs to find a new one from the pool, because if it doesn't sync timing correctly the certificate for LDAP won't validate and my work laptop won't be able to login. Maybe it needs to initialize the networking stack to handle an NFS or Samba mount in my /etc/fstab which has a nowait=0, since I use it as a boot volume.

This could be a very, very long list of hypothetical changes :)

pxe boot obviously has some constraints that don't usually apply otherwise.

Presumably the dock can be detached while booted, so the kernel should also be able to handle booting and only detect at some point that the dock has disappeared

All that USB stuff ought not to be blocking though, right? since those can also be connected or removed at any time.

This isn't true. All hardware is swappable between boots. I've moved a root disk from one case to an entirely new case before, replacing literally every hardware component except the root volume but keeping the same OS. Not only is enumeration requires at boot, but event handlers for hot-swapping need to be available at all times. Users expect to be able to plug in and out USB devices, network cables, monitors, keyboards, and have it all just work.

Presumably, most faster boot OSes simply don't support as many devices. Amazon was able to get Firecracker to boot Linux damn near instantly by pretty much removing all hardware support from the kernel since they know for sure it's only ever running as a VM on a hypervisor they strictly control and won't change.

If you know for sure the devices you need, you can compile the kernel yourself and remove all the stuff you don't need and possibly even get rid of udev and just hardcode what it normally detects.

You can see what is eating up the time as you boot if you use SystemD.

https://wiki.archlinux.org/title/Improving_performance/Boot_...

I haven’t really had any issues with Linux boot times for personal machines lately. I think the people that care most about getting into the sub-second range are the ones doing cloud VM stuff, spinning up lots of micro services or whatever.

My computer takes about 45 seconds to start, which isn't terrible, but when I run a Windows XP VM on a quarter of the CPU cores I'm reminded of how long those 45 seconds are.

Even without the stupid 20 seconds of firmware I can't do anything about, XP shows me an interactive desktop in half the time Linux does. Windows 7 is just as fast, and just as fully-featured as my desktop Linux install. Windows 11 still boots faster (and more reliably) than Linux despite the overhead of Windows Defender and drive signature checking.

I don't need 0.3 seconds of boot time on my desktop, but something a bit faster would be nice. I think my machine is so slow by the death of a thousand services all starting at boot.

Have you tried running

    systemd-analyze

And then

    systemd-analyze critical-chain

To see what is slowing it down?

My system takes 10 seconds to boot apparently, with fairly minimal tuning (I boot from a USB drive so I try not to worry too much about this, hah!). But I don’t use a desktop environment or any of that sort of stuff, so it is probably not a good comparison.

Still, 45 seconds is pretty slow, I wonder if it is waiting for something that idles out; network interfaces or something like that.

20 seconds in firmware, 8 in kernel, 6 seconds waiting for systemd networks to load, a second for Docker and an encrypted LUKS drive are most of my boot process (and a thousand tiny systemd services, of course).

I can disable a few things here and there, but whatever hardware detection takes place in the kernel (I assume Nvidia related things, it's always Nvidia it seems) isn't really somethint I can change. I can try disabling more POST parts, maybe, and disable Docker on boot to take off another second, but to shave everything down I'll probably need to reinstall.

Your original post (which, I assume, was just kind of an off the cuff guess, so I’m not trying to “gotcha” here), had us at around 25 seconds in kernel+user. So the goal of half that is around 12.5 seconds. And you measured 14 seconds. So it seems like it is in the right ballpark? I can save about a second by disabling NetworkManager… I’m sure there’d be some way to defer it, but it is not worth playing with it IMO.

Out of curiosity did you do some tuning or was the boot just faster than you guessed (if it is the latter, let’s all celebrate the fact that our computers have gotten fast enough to confuse us, haha! When XP actually came out I was a teenager with junky hand-me-down hardware, 45 second boots would have been a dream…).

systemd-analyze plot gives me the following numbers:

Startup finished in 17.642s (firmware) + 9.467s (loader) + 20.168s (kernel) + 12.175s (userspace) = 59.453s graphical.target reached after 12.102s in userspace.

Subtract the slow motherboard firmware (17.642s) and me typing in my password (about 5s, nowhere close to the 9s+20s that the loader+kernel is taking to boot before systemd kicks in) and I get about 45 seconds of "this is what Linux and tools is actually doing before I can log in".

All Windows version from XP up to 7 were horrifically slow until modern SSDs came along. For years, you could take any Vista (or even XP) era computer, double or quadruple the RAM and insert an SSD, and it would feel like a completely new machine. You can still pick up a second hand high-end Windows 7 computer for cheap and use it for what you would otherwise spend $500-700 dollars on after ripping out the hard drive and installing an SSD.

Having used Windows 7 from a HDD for years, I can tell you for sure that Windows 7 sure didn't boot this fast when I first got it :)

Yeah, that’s way too long. My laptop (intel gen11) spends ~7s on firmware, a while in kernel (can’t tell how much of that is me typing my excessively long LUKS passphrase; last boot was 11s, of which waiting for me to type was probably at least 5s…), and just 2.2s in userspace before reaching graphical.target (i.e. I get a GDM prompt). If my assumption on my typing speed is right, OS takes no longer than firmware + bootloader.

(~0.2s is wasted on a dependency that shouldn’t be there, but I’m not prepared to resolve that)

My laptop beats my PC in boot times, but also has a tendency to freeze/panic whewn switching to GDM, and when I force reboot it'll freeze a few times in a row and then work again (it's either an Nvidia problem or an Intel problem, but I don't get any stack traces because the display is frozen). 10th gen laptop versus 7th gen desktop seems to make quite a difference.

Linux definitely can be fast, but in my experience it starts slowing down as you install more services and tools, and I'd need to reinstall and reconfigure from a clean slate to get back to normal speeds. I can make it quick again ny disabling a ron of services and features, but that just slows me down later in the process.

> Linux definitely can be fast, but in my experience it starts slowing down as you install more services and tools

There's a weird tendency to put things into multi-user.target to have them run at boot, even if they are not actually required for user login. There's no actual need to start openssh before gdm, but that's how distros set things up.

That's true, but I don't think GDM actually cares about OpenSSH.

On my machine (which I didn't alter from Ubuntu defaults) GDM runs after:

    system.slice switcheroo-control.service dbus.socket plymouth-quit.service console-setup.service cloud-config.service gpu-manager.service rc-local.service systemd-journald.socket systemd-user-sessions.service basic.target plymouth-start.service getty@tty1.service fwupd.service sysinit.target

according to systemctl show gdm.service --property=After

cloud-config is the weirdest one, but it has to do with locales and system configurations for automatically deployed systems (and after first install), I believe. getty@tty1.service is also strange, but doesn't even show up in systemd-analyze blame.

In my boot-analyze charts I don't really see that many problems with illogical start orders to be honest. systemd-networkd-wait-online seems to hold up a bunch of networking services but that depends on the rest of my network responding in time so I can't even blame it for that.

Idk your specifics, but modern Windows also essentially hibernates a clean memory state image before login and resumes from it anytime it does a cold boot (it discards it and does a “real” boot when you explicitly Restart). So that seems to be one way it gets a leg up in this contest.

I know, but I've disabled most fast boot tricks because rebooting into Limux became an issue.

It's a cool trick that reminds me of Firecracker. It would be nice to see Linux use that trick as well, but with the existing issues for Linux ×+ hibernation + lockdown mode, I think it'll take a while before that's finished.

OSes not tailored for generic platforms can be built without all that conditional code that executes whenever this or that chipset is found; then all those delays because you don't know how much time a chipset or peripheral will take to initialize, etc.

Read up on “raspberry pi fast boot” and you’ll learn a lot about the tricks that can be used for fast booting, and also why they aren’t the default.

Reply on news.ycombinator.com