What are some Unix design decisions that proved to be wrong or short sighted?
reddit.com
reddit.com
“Short-sighted - users and groups.
It makes sense on a single-host system being shared at a university. It is weak for pretty much any other setting. It acts as a distraction in pretty much every setting where we now use unix: on personal systems, on cloud deployments, in the enterprise. There are constant hassles of aligning users and groups between systems, and ensuring that applications are structured along those lines.
To offer an alternative. We would be in a better place if unix provided some kind of standard process-group sandboxing, along the lines of jails. Permissions would be applied at the jail level rather than the filesystem level. The sandboxes in smart-phone OSs hint at the way.”
I feel this pain in some cloud deployments, but I’ve never thought about the whole users and groups concepts as the core of this issue.
And while there are ways to improve this with LDAP and things on Linux covered by nsswitch and so on, they just aren't very nice. And because so much is driven by uid internally finding that one system out of 100 where a constraint is violated is needlessly troublesome, compare this to Windows and AD where at any moment there is always a clear source of truth.
Cgroups fixed that in the local space but just having a clean logical mapping that is consistent across all implementations where things are wholly separate by default and there is no bleeding in of the parent OS like a VM does
Seriously, see these slides by Rob Pike from his talk in 2001: "The good, the bad and the ugly: the Unix legacy":
http://herpolhode.com/rob/ugly.pdf
Pike writes: "What makes the system good at what it’s good at is also what makes it bad at what it’s bad at. Its strengths are also its weaknesses. A simple example: flat text files. Amazing expressive power, huge convenience, but serious problems in pushing past a prototype level of performance or packaging." And so on, with several other examples.
This was understandable in a time when disk space was scarce, but it causes all kinds of dependency hell when you want to install programs that need different versions of the same library, or when you upgrade one of those libraries and break a bunch of programs.
Plan 9 got this right with static linking a long time ago: https://9p.io/wiki/plan9/why_static/index.html. Sadly this has still not been adopted by major Unixy operating systems. Flatpak and Snap seem promising for isolation of dependencies in Linux without requiring full containerization.
Compare that to trying to figure out which of your statically compiled applications happen to include the affected version of the library.
As with so many choices we need to make in software development, it doesn't distill down to one way simply being better. It's a trade off, and in this case it's not just for disk space, but also for maintainability and easily discovering how exposed you are with respect to security.
People manually build stuff. People also manually install binaries than have their own dependencies (even installed wireshark for Windows? I know for a long time, libpcap was installed separately as a sub-process during install). For example, probably 99% of the people that build their own toolchain for development and pushing to production. People also install separate package management systems (brew, cygwin) that might not have their update mechanisms automatically run, or are infrequently run.
This is a compilation and deployment strategy downside. You can mitigate (but not eliminate) this downside through a strong package management system that tracks all this (BTW, is there a package management system in current use that tracks statically compiled dependencies?), but only if users and administrators adhere to strict practices.
There are plenty of package management systems in current use where linking is an ideological or historical choice and has no relevance to security updates.
With dynamic linking, you may get updates for programs that need them without knowing that it was needed. If you manually build Nginx with shared libraries, and later on some fix for PCRE comes down the line and you update libpcre, your Nginx install takes advantage of that the next time it restarts. This happens whether you manually updated PCRE or a package management system does it for you. This happens for things like Chrome and Firefox too, even though they ship with quite a bit statically compiled. There are still portions they expect the system to provide (if you've even tried to get chrome working on an older system, this becomes obvious).
This is an advantage, it exists, and people make use of it all the time. You can try to argue down it's relevance, but without successfully doing so first, I think saying it's pointless is getting ahead of yourself.
> There are plenty of package management systems in current use where linking is an ideological or historical choice and has no relevance to security updates.
At the very least it would have relevance in that they would need to accurately track package dependencies and would not have the additional benefit of the package failing to install if they got it wrong. I've had plenty of times in the past where package management systems haven't even gotten that right, where everything installs but there's errors about missing libraries because of the way things are loaded on demand for some systems. There are benefits to not relying on some automated or manual tracking of this, where instead if you get it wrong it doesn't work. The fact that it requires a package management to mitigate this problem is itself a downside.
To be clear, my argument is there are both upsides and downsides to dynamic linking, and that it's not purely all negative. By arguing against that you're positioning yourself as stating that it's not a trade off, so it's all bad (or all good, but that doesn't seem to be your point). Is that really the position you intend to stand on? It seems rather absolutist.
What about keeping libraries dynamic, but allowing different versions to co-exist? (basically, it boils down to considering different versions of the same lib as independent libraries)
If this worked, developers would have no incentive to upgrade library versions that are working properly. The number of programs on your system that share the same library version would decrease over time until most of them use different versions, so you will miss out on the advantage of dynamic linking anyway -- updating a library would only fix security holes in a subset of your programs. You might as well just statically link everything and call it a day.
Please don't. Especially if I don't have your source code.
A reduced amount of binaries to upgrade is only the tip of the iceberg.
Even if I have 10 programs using 10 different versions of the same library, dynamic linking is a better choice than static linking.
Because it allows security upgrades, independently of any collaboration with the vendors of these 10 programs.
Similarly, increasingly many programs now ship an entire VM. Eg Eclipse has shipped with its own JRE for years now. Every Electron app comes bundled with a browser and a JS runtime, Dropbox includes Python and half its battery pack, and so on. Strangely, .NET is the odd one out, still requiring system wide installation. Well, at least it has a good backward compat story.
Sure, this makes installers huge, but nobody really cares about that anymore. Similarly the potential space and cache efficiency savings of two programs sharing the same iconv.dll is just not worth the headache.
Command line "package managers" like https://scoop.sh take full advantage of this trend by downloading these installers and not even running them. Instead, scoop just extracts their files into a directory somewhere deep in the homedir (= no admin rights needed) and assumes that said program is fully self-contained. In my opinion, this is lovely.
It's become my firm opinion that the concept of "installation" of applications is silly, causing far more trouble than it is worth.
.NET Core allows both system-wide side-by-side and local deployments of the framework.
Regarding side-by-side libraries, there is WinSxS
The reason dependency hell arises is because there is a lack of discipline in setting the soname correctly according to a semantic versioning (i.e. someone change the interface without changing the soversion)
When soname is set correctly, there isn't really that much difference between /usr/lib and the GAC in Windows.
You can choose to read is as "they chose vintage memory stuff" or as "one of the mistakes is what we might now think of as vintage memory stuff". It's all about context.
Another concept is a certain separation between users and programmers, in both LispM, Xerox Alto and even Plan9 users can smoothly program their environment easy, so users have the maximum flexibility possible and a usable a effective systems thanks to high level of integration and freedom. As actual comparison try Squeak, few Scheme implementation, Emacs itself with elisp config they are not, of course, integrated and effective like ancient LispM or Alto however demonstrate enough how powerful can be such paradigma.
Last thing the everything is a file that's really limited in unix, well implemented in Plan9. Everything is a file is a super-nice concept, but everything must be really everything.
ifconfig | grep 'inet addr:'| grep -v '127.0.0.1' | tail -1 | cut -d: -f2 | awk '{ print $1}')
(grabbed from first SO question I could find)
Text streams were great 40 years ago, but you run into the limitations of what you can do properly with awk and sed pretty quickly.
Passing structured data through pipes using a standard format would have been a universal interface. Text streams are really just a void* pointer. Sure, you have the data, but good luck making sense of it.
01 SOH(Start of Header) 02 STX(Start of Text) 03 ETX(End of Text) 04 EOT(End of Trans.) 05 ENQ(Enquiry) 06 ACK(Acknowledgement) 21 NAK(Negative acknowl.) 22 SYN(Synchronous idle) 23 ETB(End of trans. block) 28 FS(File separator) 29 GS(Group separator) 30 RS(Record separator) 31 US(Unit separator)
You've got everything you need there to build really useful serialized data pipelines, and it would only be a day's work to implement them as a library in pretty much any language.
Processing structured data is great, but like a lot of complaints here, it's basically "why didn't they just choose this much more complex and time consuming option that was as yet unproven?"
I also think that file names should be more constrained. If we didn't have whitespace in file names we could have avoided uncounted hours of people trying to get quoting right.
The former I'm not so sure. It's very 'ascii-centric', for lack of a better term. I think in order for that to make sense you'd need to disable UTF8 filenames entirely; the filesystem doesn't know, or care, that there's a relationship between a and A and any number of other Unicode variants, and why should it?
I think it's Windows and Mac OS X that made the mistake, with case-insensitive filesystems. How do they handle I and İ?
The only serious problem I've encountered, which I'd imagine is a real deal-breaker for some, is that all of Adobe's products refuse to install on a case-sensitive filesystem.
IntelliJ editors also show you a warning telling you to add the following custom property: `idea.case.sensitive.fs=true`. Although I find it odd that they make you manually set the value, since they can clearly detect when it's needed...
On rare occasions homebrew will fail to install something, but it's usually fixed quickly with a small tweak to the formula. Ever since I switched a couple years back I've probably had to fix around 2 or 3 obscure packages, and I always push the change upstream.
I think only NUL and forward slash are disallowed, which is needlessly over-permissive.
I want stdio to be fd0 and it to be both readable and writeable. I want it to be socket-like. Read from "stdin", and you read from fd0 and it's a socket. Write to "stdout", and you write to the same fd0. It would simplify child process handling code since you can use one call to `socketpair()` for stdio. Stderr could be implemented on top of the out-of-band stream, and for program-to-program communications on Linux with a domain socket, you can pass file descriptors and discover the PID of your peer.
To make this work well, I think you would need the concept of a pipeline at the file level. Eg, if process A has a pipeline file pipeA, and process B has a pipeline file pipeB, you would need some form of linkPipe(pipeA,pipeB) syscall that causes in in buffer of pipeB to be the same as the outbuffer of pipeA.
Granted, this breaks the general model of a file representing a single block of bytes that both reads and writes operate on, but that model is already gone in UNIX (eg, the in and out buffer for sockets are seperate).
I'm not convinced this is actually a good idea, but seems doable.
It's pretty easy to use a Unix socket for what you were describing.
When I think of pipes, I think of an operating system concept of synchronized DMA between unique address spaces. When I think of pipelines, I think of what you're talking about, a sequence of programs handling data between them.
I don't see why a pipeline should be unidirectional though.
Fair enough.
> I don't see why a pipeline should be unidirectional though.
Piplines usually only carry material from one side to the other :) (yes, pneumatic ones are different :)) It's part "just because that made sense at the time" and partially because it does make sense: Give data to A, take the output and process it, give that to C, &c.
However, a bidirectional, or even multiple, pipe would have some advantages, such as signaling.
Granted, this is done by default in every modern distro I have seen.
I'd say a clear failure of the unix legacy is that file creation time wasn't tracked. You only had atime, ctime, and mtime.
On macOS you can only access st_birthtimespec with stat(2) if the `_DARWIN_FEATURE_64_BIT_INODE` macro is defined.
A quick search shows most newer filesystems support birth or creation time:
* Btrfs: otime * ext4: crtime * UFS2: st_birthtime * ZFS: crtime