The extra file size is not much of an issue, but it does mean we still require stupid workarounds with Git and anything else that needs to handle cross-platform plaintext.
UTF-16 is my favourite to complain of: created because a few large companies had put a couple of years of heavy investment into UCS-2—which was a massive breaking change itself—and didn’t want to let that go to waste (with all the bother they’d been putting users through), even though by the time they actually made UTF-16, UTF-8 with its ASCII compatibility had been invented. And so they ruined Unicode for everyone for ever after with that accursed abomination surrogates and the distinction between code points and scalar values.
Or more general, the "hack" is to act as if ASCII was a character encoding, when it's actually the encoding for a set of commands to a machine.
It's also document encoding (there's characters to encode structured data too).
Given the era ASCII was invented, I wouldn't really call it a hack. Character encodings were one of the lowest level abstractions. The size of a byte wasn't always fixed to 8 bits and it was the number of bits in your character set that often defined how big your byte was (yes, I know ASCII is technically 7-bit but one of the original considerations was that the 8th bit would be a parity bit: https://en.wikipedia.org/wiki/Parity_bit ). It's also the reason the lowest size of addressable data is a byte / char.
Bare in mind that we are talking about an era before operating systems as we recognize them. Back then each machine would have it's own operating system and some institutes would even have different operating systems despite having the same machines. Really these operating systems were more like firmware than what we have today. So there wasn't any distinction between character encodings and machine commands but there were differences between machines. So it made sense that ASCII was cross purpose because that's exactly what characters were: raw addressable data containing instructions. Those instructions were sometimes printable data, it was sometimes formatting data, and it was sometimes control data.
I doubt any designers of ASCII could have even imagined the way we use computers these days when writing that specification. Even just the fact that character encodings have become an interchangeable abstraction for rendering text would have blown their minds. Never mind everything else.
The way machines worked back then would be you'd address a byte of data, that byte contained instructions. Those instructions were sometimes printable data, it was sometimes formatting data, and it was sometimes control data.
It wasn't until operating systems became more sophisticated, hardware become more sophisticated and the two became decoupled, that character encodings became the abstraction for rendering text that we think of them as today.
> In 1901, Baudot's code was modified by Donald Murray
> The Murray code also introduced what became known as "format effectors" or "control characters" – the CR (Carriage Return) and LF (Line Feed) codes.
(source: https://en.wikipedia.org/wiki/Baudot_code#Murray_code)
Thanks for the correction there. Learned something new today :)
Python 2 to 3 all the things, I guess? You’d then need another workaround choosing between CR (now obsolete), CRLF, LF and RS (renamed to LE), the latter being a true newline.