There are many ways to fail to read a file in a C program
colinpaice.blog
colinpaice.blog
It's not all roses on unix or rust side either. In unix filenames are not utf-8, which leads rust having fun things like OsString.
(Python's approach is that os.listdir("/") gives you a List[str] (silently omitting undecodable entries), while os.listdir(b"/") gives you a List[bytes]. That is, if you give the path as a utf-8 string it returns utf-8 strings, otherwise it returns bytes.)
The reality is that file names are "stringy" in nature--people expect to be able to do display them--and that means you need to have some up-front agreement on how to interpret those strings. In practice, on Unix systems, everyone has generally agreed that this is UTF-8, to the point that trying to not be UTF-8 generally causes interesting breaks in the system. It would be great if we could actually get the operating system to help enforce these rules, rather than placing the blame on other software for not correctly handling situations where the correct solution is itself incredibly ambiguous.
So I am afraid we can either a) indulge ourselves in wishful thinking, b) actively try to extinguish platforms that don't match our ideals, c) make an effort to be actually cross-platform, and not in "let's just build a tiny Linux model in a bottle for us to use and pretend the rest of the environment is not there" kind.
For untar'ing a tar archive: error out by default, but provide a flag or option to permit untarring using some sort of escaping to map the malformed names back into Unicode. I think here I'd map to something printable, though, like "\xnn" or something.
Paths might just be strings, but they aren't necessarily, and since Rust actually cares about types if you want strings you need to write the code to decide what to do about this, even if your "It's not UTF-8" case is just "Give up I can't be bothered".
Rust gets this almost right. Python gets this very wrong.
Having worked in both, I'd say they both chose ideomatic solutions:
Rust: I can't prove this is utf-8, so if you want to use it as utf-8 you'll need to tell me what to do if it isn't.
Python: if you're in the common situation where everything is utf-8 and you want to just work with strings, go do the simple thing. Or you can be explicit about wanting to work with bytes, and that's good too.
(Though I think Python should throw an error instead of silently omitting non-utf8 files.)
>>> b'\xc0'.decode('utf-8')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xc0 in position 0: invalid start byte
>>> open(b'\xc0', 'w').write('foo')
3
>>> os.listdir()
['\udcc0']
>>> open(b'\xc0').read()
'foo'
>>> open('\udcc0').read()
'foo'[1]: https://peps.python.org/pep-0383/ [2]: cf. https://github.com/bup/bup/blob/master/DESIGN#L667-L729
You can actually reconfigure Python to throw an error instead of using a surrogate escape, but only (I think) changing something at compile time: https://docs.python.org/3/library/sys.html#sys.getfilesystem...
That said, I actually didn't realize that os.listdir() silently omits un-decodable entries, which I find mildly alarming. This behavior isn't mentioned in the docs (https://docs.python.org/3/library/os.html#os.listdir) and seems out-of-character for Python, which usually raises an exception by default if data cannot be decoded to text.
Are you sure that this is actually what happens with non-decodable filenames? Reading here (https://docs.python.org/3/glossary.html#term-filesystem-enco...) and here (https://docs.python.org/3/c-api/init_config.html#c.PyConfig....), it suggests that encoding errors should be handled by surrogate escapes by default on non-Windows systems.
If it ever did that, it doesn't anymore.
Includes "Modify os.listdir(str) to ignore silently undecodable filenames, instead of returning them as bytes", but not later work where apparently this was changed to use surrogates.
I think that https://rushter.com/blog/python-strings-and-memory/ is a nice reference on that.
There are also limits to how far the UTF-8 illusion can go on Windows: while on Unix and friends a path is fundamentally a 0x00-terminated, 0x2F-separated sequence of 8-bit quantities[1], on NT a path is fundamentally a(n unterminated) 0x005C-separated sequence of 16-bit quantities, and Win32 puts a varying number of layers of makeup[2] on that. Thus on Unix you must be prepared to handle invalid UTF-8 in a filename, but can expect to roundtrip any byte sequence (sans 0x00 and 0x2F), and on UTF-8 Win32 you must be prepared to handle arbitrary WTF-8[3] and cannot expect to roundtrip any byte sequence (isolated surrogates can merge, though I don’t know if UTF-8 Win32 is willing to accept such invalid WTF-8).
Note that Rust does not use the UTF-8 interfaces on Windows (neither does it use the fundamental UNICODE_STRING APIs, however).
[1] https://yarchive.net/comp/linux/case_insensitive_filenames.h... (of course, Linux filesystems have since developed case-insensitive mount options)
[2] https://googleprojectzero.blogspot.com/2016/02/the-definitiv...
Plus, you don't need to be prepared to handle invalid utf-8 in filenames on unix, the fopen call can just be made to fail if needed.
As being prepared for invalid UTF-8 on Unix, well, it depends. If you refuse to run with non-UTF-8 LC_CTYPE or to accept invalid UTF-8 in user-provided file names, I suppose that’s on you. (Though I sure hope you are not writing an implementation of rm or tar!) If you’re trying to erase or move everything in a directory, though, you’ll have to either deal with whatever’s there or at least recognize that the action may fail.
For C programmers, talking about "standard" implies a quite particular meaning.
On the other hand, record-oriented files work just peachy with ISO standard COBOL.
Iirc, the last draft before final publication is free and it's just as good.
As another user replied, there is more going on in the post than what is specified, guaranteed, etc, by the standard.
The practice of C programming in an actual system using non-standard things is important. Also, the C language does have its problems, even within the standard. However, pinning to C problems of a not-so-helpful implementation, library, system, etc, is unfair and unhelpful I believe.
I see this as a relic from older, nobler times and the language is interesting to learn about especially since it is the base of a lot of things, but if C was a sport it would be free climbing, or maybe something even more dangerous that requires a lot of skill that I can't think of now.
In Rust there are many string types (e.g. OsString, CString, String, PathBuf, ..) because the truth is that you need to know the rules that the string your program reads or creates must adhere to, if there is no type system that will enforce those rules to you. The way a properly written program has to deal with strings in the different parts of systems could be explored in an entire programming career.
Similarily Rust tends to make you handle all the errors that could occure with file I/O. This can feel complicated, but it could also serve as a reminder on how many potential errors we don't handle in other programming languages (or at least as a big questionmark how these other languages handle or don't handle these errors). Surely you could also ignore those error cases in Rust and just have your program crash, but then it was your active decision and not something that hit you out of nowhere like a bag of bricks, with the only realistic option of ignoring it and hoping it will not happen again.
If anything something like Rust gave me a much better understanding why actual good C programs are akin to art. Freeclimbing in a minefield and all that.
On the contrary, I would say C aged really well for a language which was created to support an entire zoo of computers and operating systems. It is worth pointing out that the language has progressed a lot since then and you don't have to deal with many old headaches if you write C on a modern CPU architecture.
I just think an introduction to the language in the year 2022 should at least aknowledge that the form of string handling shown in the first part of the book should not be imitated. I can see how those examples would make perfect sense in a different age. Maybe I can give the book the benefit of the doubt as it was published in 2012.
Do you have any book recommendations for a more modern C approach?
Naturally it tends to be forgotten, as most UNIX folks set the genesis of computing world in Bell Labs.
This is the case only because the standardized C was more-or-less created as a superset of the many, many C variations that have sprung up until that point. It's also the reason why C leaves so many things up to the implementation or entirely undefined.
Ultimately, this made C a highly portable language, while writing conformant and portable C programs is very difficult.
I think maybe motorcycle racing. Fast, close to the ground, and one seemingly-trivial mistake away from a gruesome result. But also a rush when it goes well, responsive to riders knowing their machines and the terrain inside and out and eligible for lucrative sponsorships.
Make your rope as best you can but once you jump it's up to luck and whether the gods are feeling beneficent if you survive, and if you do, if you still have your limbs and retinas attached.
Simply put, the book doesn't deal with alternatives to ASCII because there hardly were alternative text encodings at the time of writing in the western world, which is clearly the focus of an English book written by a Canadian and American.
Moreover, the point of the book was to propose the C programming language and showcase how the language works. It's not a book on best practices or how to use C in the real world today. There are other resources for that.
The line counting program on page 19 is correct for UTF-8 inputs. The word counting program on page 20 works as specified (the specification says it only uses 3 specific delimiters) for UTF-8 inputs. The digit counting program is correct for UTF-8 inputs. The "longest input line" program doesn't really specify what "longest" means, but it finds the one with the most bytes.
There are maybe 2 examples that don't work on UTF-8, if the standard is that "longest line" and "character counting" programs should detect that ":regional_indicator_s::regional_indicator_u:" is 2 characters while ":regional_indicator_u::regional_indicator_s:" is 1 character. Such programs may not make a very good introduction to programming though.
Define "fail" and define "Unicode".
Does "fail" mean "iterates through bytes and not characters"? Does "fail" mean "can't recognise different encodings of the same 'character'"?
Does "Unicode" mean UCS2, UTF-16, UTF-32 or UTF-8?
Because, to be honest, quite a log of 'unicode-aware' languages will "faiol" the same way, and they don't have the same excuse as 'strlen()' does, namely being 35 plus years old.
I think the fact that Unicode handling can still be such a mess in applications written on platforms and languages that came decades after C tells me that this is not an easy problem.
C handles UTF-8 byte sequences with the current string functions just fine. You're going to have to manage the mapping between byte sequences and glyphs being displayed to the user, which you're going to have to manage anyway because "Unicode" is so ambiguous. Whatever support a modern language has for Unicode doesn't help all that much when the user-facing glyph isn't part of the language.
What your language thinks is a character and what the end-user thinks is a character are two different things. C is not very different in this regard.
I dunno Z/OS, perhaps everything sucks that bad there. I strongly suspect that there's alternate interfaces available that hide this complexity from those afraid to trip over it.
You can still see the UNIX-centric point of view in stdlibs of other languages: I am particularly amused by Golang's "os" package. It's kinda-sorta supposed to be portable and OS- and platform-independent, but it's designed for POSIX-likes first which is why one has to pass 0666 or whatever as permissions when trying to open a file on Windows (even though it is completely ignored).
<Snicker>
Well. That's amusing. So everytime one does file access through Go on a windows machine, one invokes the number of the Beast, eh?
Apropos af if true. Also hilarious if it just by chance worked out that way.
[0] https://cs.opensource.google/go/go/+/refs/tags/go1.19:src/os...
https://www.ibm.com/docs/en/zos/2.4.0?topic=functions-fopen-...
I’m not a fan, either, FWIW, but those are the most frequent answer I’ve seen as to why fopen() accepts a string and not a set of flags like open(). Might be a post hoc rationalization, though,—I’m willing to believe it was originally just a hack for conciseness, for example.
[1] https://www.gnu.org/software/libc/manual/html_node/Opening-S...
BTW, fixed length records? Way old school and yuck. Just use an index into the file with lengths already. What about all the wasted space from records that aren't full?
What you would like, I think, is something like git (or IPFS) where data is stored as content-addresses blobs and metadata (including filenames and directory structures) are also just blobs in the object store.
A hierarchical data store comes baked in with an opportunity of implementing topical locality for the end user, which allows you to utilize pathfinding logic baked into your brain to navigate the corpus of information in question. Content addressable stores, require praying that the layers of cryptography work, or you have enough understanding of the implementation details and tooling around the store to find what you need.
In short, find | grep being strictly necessary, rather than a fallback, means you've failed at organizing things so your user can understand where the hell something even is, and why it is there.
I assure you, more harm is done by forgetting the fundamental human way of life that computing tries to plaster over, as we inflict impedance mismatch on Users by forcing them to search in a way that makes sense only to the machine, rather than to them.
Sometimes a little less ideal computational performance pays dividends in ease of picking up.
But that's not really what I'm getting at. I'm more thinking like POSIX API vs. Git plumbing API.
> A hierarchical data store comes baked in with an opportunity of implementing topical locality for the end user, which allows you to utilize pathfinding logic baked into your brain to navigate the corpus of information in question.
Most documents naturally fall into more than one hierarchy, and some "flat" patterns as well (e.g. alphabetical by author).
One of the downsides of computer FS is that they encourage a single name-based hierarchy (although using symlinks or hardlinks you can reference files from several directories.)
The hierarchy can and should be separate from the object store. Then you can also use e.g. Jef Raskin's Zooming UI to organize topical locality, in addition to more traditional UIs like directory trees.
And I remember that I half-jokingly proposed in some other discussion about file paths to either remove the filenames entirely or at least lift the uniqueness restriction: after all, if you have a GUI, files with the same name don't cause that much of a trouble.
ln -s /a/long_contrived path/somewhere_else ./here
cd ./here # cwd somewhere_else
ls -al ../.. #list contents of /a
cd .. # cwd back to here (possibly shell-dependent)
This is one of the many searching for data/pathing based things that trips up most users, without fail, who have better things to do with their lives than internalize the madness of software developers, necessitating people like me to clean up after you heathens when you shine a turd.
Yes. Very shiny and efficient. Still a turd.
We need to build things that intuitively map through things they are well equipped to deal with, which means sometimes stepping away from the screen for a while. Cognitive load is the enemy. Any idiot can make a more complicated system. It's about building and leveraging mental spaces, and being able to wire into a skill every other human being will share in their lifetime.
We should endeavor to have computer use not condition the user in a maladaptive way to real life. I assure you, we've failed epically, even in or brightest moments.
>“Normal files” have data in EBCDIC