Hacker News Reader

Comment by gumby | Hacker News Reader

The rant behind her post (https://drewdevault.com/2020/09/25/A-story-of-two-libcs.html ), which has had some circulation, really shows its author’s limited perspective.

glibc needs to solve two hard problems: be very fast and run on innumerable systems. Some of that conditional stuff is because all the world is not Linux or BSD; some of the macrology is there to make sure such handling is performed everywhere needed, and of course the preprocessor is the closest a language like C can get to preprocessing.

I was in the code as glibc started to exist (we paid for a lot of it) and it looked like Musl: very straightforward.

View on HN

The definition of the ctype functions as working on unsigned char values and EOF + CHAR_BIT being 8 everywhere now basically means that there isn't much locale-specificity to the ctype functions: they can be made to work with ASCII, ISO-8859-*, and... EBCDIC, but not UTF-8 in general (just ASCII) or any Unicode encoding (idk, maybe they can be made to be locale-specific for Shift-JIS, but only for ASCII in Shift-JIS).

And... yes, glibc does have support for EBCDIC, which is probably ultimately why it has these run-time indirections in its ctype. There's no other reason to have run-time indirections for ctype functions given the limitation of unsigned char values + EOF. That means this code can be simplified a great deal.

Anyways, yes, Drew DeVault's rant misses glibc's need to support EBCDIC, but glibc is exactly like this for every little thing -- an unmaintainable mess. There has to be a better way to produce a fast C library w/o being such a mess on the inside.

They obviously can be made to work for Unicode; iswhatever(x) can just report the whatever property for code points U+0000 through U+00FF, its documented range. If you want to know about higher-valued code points, use iswwhatever, the wide version.

But that is ASCII. That part of Unicode is a literal copy of ASCII. In any case, just putting an "if out of range, return 0" clause wouldn't hurt performance noticeably given all the indirection already present. If used in a loop, the CPU will predict that branch perfectly every time if your data is correct. There is no reason to just crash.

ASCII goes to 7F, not to FF; it is a 7 bit character code.

Therefore, for instance, isspace(0xA0) might usefully report true if we are in a Unicode locale, otherwise not.

The 0x80-0xFF values are also used in 8 bit extensions over ASCII, like ISO-8859 1 and ISO 8859-15 character sets. E.g. 0xE0 is à in ISO-8859 1 (which is, of course, the same as the Unicode U+00E0 but logically distinct).

A totally different 8 bit extension is KOI-8.

The point is valid that if you don't support any "weird" extensions to ASCII (just ISO Latin) or non-ASCII 8 bit, then there isn't much of a need for run-time table indirection. The cases that may arise can be handled ad hoc. Along the lines of "if we are in an ASCII locale, then report false above 7F, otherwise go through the combined Latin/Unicode combined table".

No not really. wchar_t is not Unicode. It's whatever the current locale's codeset and encoding demand. No, just say no to all of this.

Instead:

  - use UTF-8 locales
  - use a Unicode library for all things Unicode
  - make your own ctype for when it's Just ASCII

I don't know who you think said wchar_t is Unicode, but it wasn't me.

In a given locale, you can almost certainly regard wchar_t as being a continuation of the range of unsigned char. If you want to know whether the value UCHAR_MAX + 1 is alpha-numeric, you can't pass that to isalnum, but if that value is in the range of wint_t, you can pass it to iswalnum.

For values 0 to UCHAR_MAX, it would be surprising if isalnum and iswalnum produced different results.

I do get what you're saying, but musl also has to live in many different worlds. Using the example where glibc is trawling into endianness in the post you linked, for example. Musl runs on a bunch of different big and little endian router boxes and other unusual use cases. While I haven't tested, I'm guessing that their much simpler isalnum() works fine on all of them.

Musl does have a lot less legacy to contend with, and musl is often much slower than glibc, so your point stands, of course.

> While I haven't tested, I'm guessing that their much simpler isalnum() works fine on all of them.

isalnum works fine of both, it only veers off when you get into UB which is UB.

If you define “works fine” as “gives correct answers even in ub” then musl’s is completely broken since it only gives correct answers for english in ascii.

It can't give correct answers for anything other than English in UTF-8 locales.

It can't give correct answers for any non-Latin scripts in any locales.

The problem is ctype and POSIX.

Given that, making ctype only work for ASCII (and maybe EBCDIC if you're really unlucky, which glibc is) is basically sufficient.

> Given that, making ctype only work for ASCII (and maybe EBCDIC if you're really unlucky, which glibc is) is basically sufficient.

Of course, but then why complain that glibc doesn't work outsid of ascii is the point.

But that's the issue: glibc's ctype does work beyond ASCII: it works for ISO-8859-*. It's pointless because that's pretty much all it can work for.

Nothing in POSIX mandates that isalnum has to be implemented using UB. It is entirely permissible to return 0 for values out of range. Or -1. Or anything else. POSIX is not part of the C language, if a function's behaviour is not defined for a given input, that's not UB, that's a license for the implementation to do whatever is natural. Which can be crashing embarrassingly, but why would you do that.

> Nothing in POSIX mandates that isalnum has to be implemented using UB.

That is a misunderstanding on your part. Calling isalnum with out-of-range values is UB. Per-spec: "If the argument has any other value, the behavior is undefined.".

> It is entirely permissible to return 0 for values out of range. Or -1. Or anything else.

Of course it is, it's UB, there's nothing it can't do.

> POSIX is not part of the C language, if a function's behaviour is not defined for a given input, that's not UB

So a behaviour which is not defined is not an undefined behaviour. Sure. Whatever.

> that's a license for the implementation to do whatever is natural. Which can be crashing embarrassingly, but why would you do that.

Why wouldn't you? A table-based implementation is flexible, convenient and efficient, and you don't care what happens outside of the function's bounds.

musl's "isalpha" is trivially wrong, for instance it wouldn't support "ç" (0xe7) or "ß" (0xdf) in ISO 8859-1 which are both alphabetic characters which fit in an unsigned char.

ctype is trivially non-localizable to locales with codesets larger than sizeof(unsigned char) anyways. Maybe the problem here is POSIX.

oh yes, no code written in 2021 should use that mess. but a glibc being some level of posix compatibility.. hard to blame them for at least trying to make it work.

Hmm, well, I mean, if ctype can't work for any interesting non-ASCII (and non-EBCDIC) cases (no one should still be using ISO-8859 locales...)... maybe stop trying so hard?

Those both return 0 for isalpha() on glibc for me, with or without export LC_CTYPE=iso_8859_1

Is there some other setup I'd need to do to see it work in glibc?

most likely you need to build the locale on your system (uncomment the relevant line in /etc/locale.gen and run sudo locale-gen).

here

  #include <ctype.h>
  #include <locale.h>
  #include <stdio.h>

  int main(int argc, char** argv)
  {
    setlocale(LC_CTYPE, "fr_FR.iso88591");
    if(isalpha('ç'))
      printf("ok\n");
  }

prints ok (with the file in the correct encoding)

isalpha() works with the "C" locale unless you first call setlocale().

For example, on my system isalpha(0xe7) is true if I first call setlocale(LC_ALL, "en_US.iso88591").

well, yes, in "normal" C programs you're supposed to fetch the locale from the user's env vars (with setlocale (LC_ALL, ""))

> Okay, fine, my bad. My code is wrong. I apparently cannot just hand a UCS-32 codepoint to isalnum and expect it to tell me if it’s between 0x30-0x39, 0x41-0x5A, or 0x61-0x7A.

Yikes. If you have wide characters, you want iswalnum, or else preprocessing: (ch <= UCHAR_MAX) ? iswhatever(ch) : 0, assuming positive ch.

There’s nothing more off-putting to me than a rant based on a wrong premise. All that hot air and insults and in the end he was just wrong, by his own admission.

> all the world is not Linux or BSD

since when does glibc run on bsd

Well for one, Debian GNU/kFreeBSD.

didn't glibc exist before linux ? surely it would have been running on bsd then

It did! The glibc 1.09 README lists a number of supported configurations, including a couple with BSD, and has no mention of Linux.

https://sourceware.org/git/?p=glibc.git;a=blob;f=README;h=b9...

Reply on news.ycombinator.com