Hacker News Reader

Comment by WhitneyLand | Hacker News Reader

One way to think about text, is as the only data we use that shrinks over the years. Every other type of data continues to grow.

The way this can be stretched to make sense is by thinking for example how much these have grown over 20 years.

Media (music, video, images) grows staggeringly because of 200p > 4k, 128bit mp3 to flac or whatever and there's more of it all that's gone digital.

Code size continues to grow quite significantly even.

But text? Even the massive wikipedia's size, only a fraction is text content.

Then as memory capacity also grows it's really a back and forth battle between computing resources and all other data, except text.

Hard to think of many places 4 byte chars would matter nowadays, maybe the tiniest devices for a few apps?

View on HN

Conversely as CPUs get more and more powerful the overhead of a non-fixed-width encoding also becomes negligible unless you're processing a huge amount of text. And if you process enough text that it becomes a problem then maybe size would be an issue too.

Furthermore 4-byte chars also have the endianess problem of UTF-16.

I think the rule of thumb of "use UTF-8 unless you really have a good reason not to" still holds true today.

I think that’s a valid and interesting point that the classic time versus space computer science trade-off needs to be considered in this case for large workloads.

One thing that seems different is that you also hear concern from developers about coding complexity and the desire to always have things to be as easy to reason about as possible. Yes these are things a library should just handle for you but it sometimes useful to know what’s happening fundamentally or can sometimes allow for less surprises.

Nevertheless, if one of the cases you mentioned became significant I’m not sure that not many would say coding complexity should take precedence over your considerations.

I agree with you there, I think in the vast majority of applications these days when it comes to text processing simplicity and robustness beats other concerns.

It's unclear if UTF-32 beats UTF-8 there however. UTF-8's encoding is a bit trickier to deal with of course, but it's really not that bad. On top of that you have a few advantages such as not having to worry about endianess as mentioned previously but also not having to worry too much about alignment or corrupted input (you can re-synchronize UTF-8 at every codepoint since the leading byte has a unique bit pattern). And of course ASCII/C-string compatibility means that it's unlikely to be mangled by non-unicode-aware applications.

In any case the complexity probably won't be in the encoding, be it UTF-8, 16, 32 or anything else. UTF-32 might be constand-width codepoint but in general you don't really care about codepoints since they can be combined in various ways to create characters or things like country flags or complex characters with diacritics.

> [...] when it comes to text processing simplicity and robustness beats other concerns. It's unclear if UTF-32 beats UTF-8 there however [...]

I always wondered how this affected simplicity and efficiency of different text editor implementation strategies. I'm mainly thinking about the variable byte lengths and how it affects anything related to character indices. I'm guessing if it seriously hinders a preferred strategy one would just convert to a plain codepoint of fixed word size internally at the cost of greater memory usage... which actually is not that bad if you only want to support the BMP without any fancy encoding which would only takes up 2 bytes internally.

СPUs aren't getting more powerful. Haven't been for years.

How do you figure that?

String-keyed hash maps are really common, and the keys tend to be ASCII. This would increase their size by a factor of 4, resulting in substantially increased memory usage and decreased performance.

Certainly, but I’m not sure myself, can you think of any scenarios that are common where the difference would be noticeable or significant?

Boot any modern OS and you’ll have megabytes if not tens of megabytes of these strings, and spend a small but significant amount of CPU time looking them up. This may not matter on your Ryzen desktop with 97 cores and 318GB of RAM, but it’s substantial on a smartphone.

Well, that seems to be an unfounded assumption that I'm considering only powerful machines, especially since possible challenges on small devices were mentioned in the OP.

In fact, I done a fair amount of ARM coding so I understand what you are trying to say.

Nevertheless, your comment seems to point out smartphones may be less powerful, but sheds no light on what the delta might be for boot performance or how you can be confident it's say, more than a few hundred milliseconds.

Maybe it is, I haven't isolated and investigated that single difference enough to form a reasonable guess. Have you?

Any dynamic language, as all the symbols are strings.

It's not just that the raw encoding that becomes easier to relax with higher system resources - it can also be generically compressed using those same resources. I run my Windows machines with disk compression enabled, and that tames most "bloated" files and lets me fit a lot more into a 256Gb SSD.

However, there is some appeal in defining the format to a fixed low fidelity, too, since it shapes the authoring process around conveying the content through the limitations - "the medium is the message." But they're more likely to be the source format than thr transmission format, in the same way that old video games designed for 8kB of working memory now end up with terabytes of video footage streamed to online audiences.

Nitpick: it’s not shrinking, probably growing faster than ever, it’s just growing slower than other forms of media.

Fair enough. Agreed, it would have been expressed more clearly as a some sort of relative perception.

Reply on news.ycombinator.com