Hacker News Reader

Comment by nickcw | Hacker News Reader

Just in case anyone is wondering why we had (or used to have) /bin and /sbin directories as well as /usr/bin /usr/sbin here is my understanding of it.

It is because on old Unix systems there frequently wasn't enough space to store /usr on the root disk. Therefore /usr was a separate disk which might not be available during boot. This meant that you needed to put all the binaries you needed for boot in /bin and /sbin because /usr/bin and /usr/sbin might not be mounted yet.

This no longer makes sense in the days of large disks which fit /usr just fine and initrds which can mount all the disks before kicking off the things which might need those binaries. In fact I think the initrd has taken the place of /bin and /sbin on a modern system containing copies of all the binaries needed to mount the disks (like mount and fsck).

View on HN

> It is because on old Unix systems there frequently wasn't enough space to store /usr on the root disk.

It was really a single system in 1971 that kicked off this trend. Originally /usr was for user files, like /home is today (do you'd have /usr/dmr, /usr/ken, etc.) / and /usr were two physical disks, and at some point the / disk (a 1 or 2MB disk IIRC) was full to they put some things from /bin in /usr/bin as a hack. And it's "stuck" since then. This is why /usr has such a weird name, because it was intended for user files (/system, /software, /additional or something would make more sense, abbreviated to /sys, /sw, or /add of course). Initially Unix was a quickly developing system for internal use, but it was actually used and there was often some tension between "do the right thing" vs. "don't break too much". Some of the weird things in C are due to that as well (e.g. &'s precedence being a good example).

I guess some other systems that ran Unix ran in to similar problems, but by the time unix started to gain some adoption in the mid to late 70s disks were larger too, so I don't know if it was ever really needed beyond that initial research unix system used only internally at Bell Labs. I believe they updated the disks on the original Unix system not too long after this making this hack superfluous, but they kept it for compatibility (and because it might be useful again in the future).

Is that also where /usr/share got its name for application data? Data that's shared among the rest of the users, as opposed to in their respective homedirs?

share is for architecture-independent data (ie data which is “shared” by all architectures). Executable binaries are not shared, but things like man pages are.

I believe it dates back to the heyday of diskless workstations. It was commonplace to have dozens of workstation machines getting their OS files over NFS from a central server.

Thus why the (per-machine, read-write) /var became standardized. Earlier /usr contained a mix of writable things (like /usr/log). Now it was important to make /usr something that could be NFS mounted read-only by a bunch of NFS clients.

There were also UNIXes that ran on multiple architectures, like SunOS on 68k and SPARC. In theory, you could have them both use the same read-only /usr/share mount and save some resources on the server.

I say "in theory" because I don't know how many sites ever implemented this in practice. There were definitely places that supported mixed-architecture diskless fleets, but I'm unsure how many of them were committed to keeping the OS version pinned between them... or how many then went through the extra work to make /usr/share a separate mountpoint. I'm sure some people did it, but it's not something I remember seeing personally.

Still, I feel it's somewhat useful to help humans understand what parts of a package are CPU-specific and which aren't. Probably good that it's lived on.

I think /usr/share is linux specific. so much later

It exists on BSDs as well. (No idea who did it first.)

I think the first place I saw it was on SunOS 4, but not sure if that was the first. Definitely existed pre-Linux

Around 1990 I used Unix clusters. Not only 68000 Sun mixed with SPARC, but also the other Unix workstation vendors. There was a lot of NFS and also completely diskless machines. /usr/share always meant it's architecture-independent and can be used by machines of any architecture.

I am so happy to see that this correct explanation has finally spread.

When I first started posting on HN the standard response you would get was the retcon explanation like the parent comment or even more nonsensical ones like "usr == UNIX system resources".

At the time cursory searching could not find any actual explanations for this. I started a deep dive into Usenet history and reaching out to folks who knew the history, looking for scraps of info in old Unix books, etc and found the correct explanation as you posted. It was entirely an ad hoc solution to running out of disk space. /usr got moved to the second disk. When root got full again they moved some large binaries to /usr/bin. When the /usr disk got full the user home directories were the easiest thing to move so they moved to /home on a third disk. Every other explanation is purely post-hoc rationalization.

Since then I've been pointing it out when the topic comes up and I am happy that general knowledge of the actual history has started to spread.

Note: I am not taking credit for this, many other people have been correcting the story both before and after me. I'm just glad the true story is seeping into the collective tech community. Sometimes misinformation or incorrect "facts" seem to linger no matter how many times and how many people correct them... but this makes me hopeful that misinformation can be corrected.

Your comment makes me want to make up some new nonsense and try to spread it in here. I'll restrain myself though :)

It should also be noted that, while this change is Linux-specific, it does not directly break software which also targets BSD or nix-like OSs.

Files that were previously in /usr/bin or in /bin can now be found in EITHER of these locations, since one symlinks the other. So no previous expectation was really broken.

Only software built on merged systems fails to run on unðmerged systems. This should not really happen, since the usr merge was a one way trip, not a flag that you're supposed to turn on and off. You'd also never build dynamically linked binaries for BSD on Linux, so that should not be an issue.

But, for some reason, Debian chose to make this merge something that individual systems could turn on and off, which is a terrible idea for part of the base system. It's like letting users pick if they way `/bin` or `/binaries`. Having such heterogenous setups in regards to something so basic and foundational is asking for breakage.

Debian didn't "choose" anything. It's just a bunch of people who wanted merged-/usr to become reality created a tool to convert a system until a project-wide decision is made. The decision has been made ultimately, but the dpkg maintainer blocks it from being implemented.

> Files that were previously in /usr/bin or in /bin can now be found in EITHER of these locations, since one symlinks the other. So no previous expectation was really broken.

I don't know, I just hit breakage the other day. I have /usr/bin before /usr in my path (which is the default on Ubuntu at least); I have muscle memory to use dpkg -S `which $foo` to figure out which package a binary is, but that doesn't work if dpkg thinks the binary is in /bin (e.g. ping), since it'll ask dpkg who installed /usr/bin/ping, which is nobody.

It is small fiddly things like this all over people's packaging and personal scripts that break.

This is a _very_ clear P.O.V.:

Who installed '/foo/bar/baz' when '/foo' is a symlink to '/usr/bin'?

I'm 100% in favor of the DPKG maintainer's perspective of "do ugly symlink farms" and then "reap what you sow" (ie: if you don't like there being a symlink there, then fix the offending package).

Adding to this historical reasoning, it is to be noted that in traditional Unix, /usr was pretty much the equivalent of /home in Linux systems: the parent directory for user home directories, i.e. 'usr'.

The existence of /usr/bin can probably be explained by the reasoning that, just like nowadays Linux systems often have a really big /home partition, early Unix systems eventually ran out of space for whatever volume would hold /bin, and with 'reinstalling' not remotely as easy an option as it would be these days, storing an extra 'bin' etc. directory in the 'larger partition for user directories' became commonplace.

Careful, you'll attract computing historians who will want to tell you that "usr" actually stands for "UNIX system resources"

That's how I've learned it around 1995, and that the division between /[s]bin and /usr/[s]bin has the reason that the former are required for system boot, while the latter are mounted via NFS from a central server - why waste local disk space when these would be identical for every system anyway (in a homogenous environment, at least). While that made a lot of sense to me back then, I've become aware that it might not be absolutely accurate, though.

NFS is a Sun thing. If you consider Sun as the canonical historical Unix, this is still correct.

Sun had a few interesting early views on networking.

Nice backronym, but /usr/ken (etc.) in v6 shows that /usr was originally a user disk.

You could ask Ken Thompson (/usr/ken) himself to verify. ;-)

Given that he is THE Ken Thompson, I'd argue that he was, in fact, the UNIX system resource.

Bahahaha. Same probably goes for everyone else in /usr/ like /usr/dmr.

FreeBSD's hier (7) has some good background info on older Unixisms https://www.freebsd.org/cgi/man.cgi?hier(7)

Of note, though not called out in that link, FBSD uses /usr/home/foo as the home directory of foo, not /home/foo (though they are often symlinked)

There was a long thread about this 10 years ago: https://news.ycombinator.com/item?id=3519952

Old Unix guy here: That is my understanding as well.

But then the question becomes, why not get rid of /usr?

Having the "immutable" parts of UNIX reside in a single directory is often useful. It can be mounted read-only, for example.

/usr/local is not immutable, nor is /usr/share .

Also, you could mount / as read-only, and mount the writable parts at different points, or using a UnionFS mount of / .

> /usr/local is not immutable, nor is /usr/share

Not technically, but I've built a number of "appliance" linux systems for clients, and to improve reliability I just make the entire disk read-only with the exception of /home and /var.

The few locations outside of /var that sometimes need to be writable (in particular, /media, /mnt, and sometimes /root) can simply be symlinked into /var.

> why not get rid of /usr?

To avoid breaking backward compatibility.

You could have links from /usr instead of from / , couldn't you?

I think the path of least resistance is symlinking /usr/bin/ to /bin, but now that I think about it I think I misunderstood GP's suggestion.

GP's idea, as I am now thinking about it it, is that you could basically move everything out of /usr into /, effectively getting rid of /usr.

Symlinking /usr to / seems like a dubious idea (since we'd get weird things like /usr/etc/passwd) but turning all of its top-level directories into symlinks seems like a possibly OK idea.

Looking at my ubuntu installation, /{bin,sbin, lib,lib32,libx32,lib64) are all links to /usr/{...}, which seems backwards to me. I think they should have hoisted everything into / and made /usr/* symlinks for backward compatibility.

macOS still has a traditional BSD style /bin (37 utilities) and /usr/bin (1000+).

(But why is /bin/sleep a 150K executable? maybe it's a fat binary in more ways than one?)

This.

I just don't understand, if people want to merge /usr and /, why they insist on keeping the gratuitous /usr prefix. It can all just be rolled up into / : /bin, /sbin, /share, /lib, /include, /share, /src and so on.

The obvious followup question that jumped to mind is "what's the difference between /bin and /sbin?".

Both contained files required to boot in single user mode, however /sbin contains executables not usually executed by normal users. sbin short for "System Binaries" maybe?

I always understood it as statically linked binaries that could be used regardless of the state of the rest of the system.

Both /bin and /sbin are generally statically linked. So you're not wrong; but primarily sbin is intended to mean "executed by root" above all else.

Looking through my /bin and /sbin, none of the files are statically linked.

Hrmm what distro? I haven't looked in sometime, maybe something changed (Slackware here). My understanding was static linkage in boot related folders was to minimize chance of accidentally losing a library/dependency and breaking booting.

Looked at both Debian and Redhat.

How can one discern whether a file is statically linked?

file(1) or ldd(1)

I understood it to be 'secure binaries', locked away from regular users through security policies.

This is not correct.

Separate /bin and /sbin were so that Unix systems could start with a small, minimal, clean root disk that could be read only or effectively read only. The /usr partition was sometimes mounted separately, but this was more for proper organizational/systems design division than it was for disk space reasons.

This is still a useful division in many Unix systems today (e.g. IOT) where starting from a known-good minimal image and then layering filesystems on top can help reduce complexity and prune the debugging tree.

It's not surprising that the ibm/redhat/systemd/freedesktop crowd doesn't care about this stuff, but it's unfortunate.

Rob Landley doesn't agree:

http://lists.busybox.net/pipermail/busybox/2010-December/074...

Peace to Rob, but disagree with his recollection and the motivation. This was a very active topic in the early commercial Unix days as POSIX etc. were being formulated. It was very common to keep a tar backup of / and to separate out your /usr mount not because you didn't have space on /, but because you wanted to be able to lose/upgrade/etc. / without impacting /usr, and perhaps to move/copy /usr to another machine. The separation of concerns, not the disk space, was a driver for why we did it that way.

Both recollections can be correct. Sometimes an idea that starts for one reason is kept for different reasons.

I started using Unix (as a sysadmin and systems programmer) in about 1986, and my recollection is closer to Rob's.

1. Never used tar(1) for backups till much later, always dump(1),

2. Never wanted to separate backups or upgrades of / and /usr, but always wanted and needed to do that for /usr/local and wherever we put home dirs in those days (which I don't recall right now).

3. The / and /usr thing was definitely explained to me in terms of disk space, more by comparison with the VMS system I was also managing which had 1MB drives. But I was too late to have "been there".

It's entirely possible that there were multiple rationales/retcons at various stages, maybe in serial or parallel; we could all be right. My 'been there' is also from ~1986 (AT&T 3b2, 300 baud acoustically coupled to an AAA-30). That said, the idea of / immutability and /usr variability was highly valued for its systems encapsulation properties for many decades, regardless of how we got there. As others in this thread have indicated, it's still considered valuable for that reason in space-constrained and security-critical domains (e.g. IOT).

Well you are wrong. All of that was a retcon after the fact. Ask the folks from Bell Labs because that's where the story comes from:

/usr meant Users. Duh.

They had one disk because disks were expensive. The disk got full, they got a second one, so they moved /usr to the second disk.

Then the root disk got even more full and they moved some binaries to /usr/bin purely for disk space reasons.

Then /usr got full but by this time it was too baked in to change so they invented /home and moved user home directories there on yet a third disk.

Union mounts and other such solutions hadn't been invented yet. It was entirely an ad hoc solution to an immediate problem. This is simply a historical fact. I don't know where the urge to retcon a bunch of justifications for it comes from.

> Then /usr got full but by this time it was too baked in to change so they invented /home and moved user home directories there on yet a third disk.

This does not align with my memory at all. /home came much later. Both AIX and SunOS/Solaris had it early, but not anywhere near as early as the / and /usr split.

Reply on news.ycombinator.com