Hacker News Reader

Comment by jujube3 | Hacker News Reader

Actually, I can just assume UTF-8, since that's what the world standardized on. Just like I can assume the length of a meter or the weight of a gram. There is no need to have dozens of incompatible systems.

View on HN

That comparison doesn't hold.

If you're dealing with lengths, you can get input data in meters, centimeters, millimeters, inches, feet, etc.

If the input data is human heights, would you automatically assume meters even if the input data is "183"?

If the input data is the weight of humans, would you always assume grams, even if the input data is "75"?

would you guess the unit instead of specifying what you expect?

> would you guess the unit instead of specifying what you expect?

It depends on the circumstances. It might be the least bad thing to do. Or not.

But that wasn't my point. I replied to this:

> I can assume the length of a meter or the weight of a gram

Sure, the length of a meter and the "weight" of a gram are both standardized. (To be very picky, "gram" is a mass, not a weight. The actual weight depends on the "g" constant, which on average is 9.81 m/s^2 on earth, but can vary about 0.5%.)

So if you know the input is in meters, you don't need to do any further processing.

But dealing with input text files with an unknown encoding is like dealing with input lengths with an unknown unit.

So while UTF-8 itself might be standardized, it is not the same as all input text files always being in UTF-8.

You can choose to say that all input text files must be in valid UTF-8, or the program refuses to load them. Or you can use silent heuristics. Or something inbetween.

> It depends on the circumstances. It might be the least bad thing to do. Or not.

It is not.

https://en.wikipedia.org/wiki/Mars_Climate_Orbiter#Cause_of_...

Is that supposed to be proof that it never depends on the circumstances?

Write all software like NASA needs to do for spacecraft?

Let me know when anyone has developed a web browser to those standards.

I don't understand the outraged tone. Asking developers to write actually good software shouldn't be viewed as some kind of crazy imposition. We don't have to write everything like it is running on a spacecraft (which I never claimed), but we should try to make it actually good. For example, if there was a web browser with a security compromise and the makers left it unfixed for a long time, there would be consequences. Saying "well, it's just a browser" wouldn't cut it.

More to the point, what situation can you think of where guessing measurement units is a good idea? In a CNC machine? Maps program? Somewhere else? You seem to have omitted the actual counterargument part from your counterargument, while adding a hearty dash of misplaced outrage.

Giving an example of a specific circumstance where it's bad doesn't invalidate the idea that it "depends on circumstances".

Why would you assume gram when the SI unit is kg and the imputed value fit?

(yeah, my only grip with SI, but it's a frustrating one.)

You can't assume it if you're handed a random sample of text files.

You can start by assuming UTF-8, then move on to other heuristics if UTF-8 decoding fails. UTF-8 is "picky" about the sequence of bytes in multi-byte sequences; it's extraordinarily unlikely that text in any other encoding will satisfy its requirements.

(Other than pure ASCII, of course. But "decoding" ASCII text as UTF-8 is safe anyway, so that hardly matters.)

Shift-JIS specifically cannot be reliably detected, and can pass for UTF-8 but actually make no sense if you looked at a conversion of it.

Do you have an example in mind? Looking at the Shift-JIS encoding tables, that seems unlikely to happen in a text of any nontrivial length; there's a small number of Shift-JIS sequences which would be valid as UTF-8, and any meaningfully long text is likely to stray outside that set.

I don't think it's fair to require "meaningfully long text" since when you're dealing with strings in programming they can often be of any arbitrary length.

Even then, I think it's likely that a your average short byte sequence that's valid Shift-JIS would still not be valid UTF-8.

I disagree... half of the UTF-8 Latin-1 supplement range overlaps with the entire half-width katakana alphabet.

Encoding detection is usually applied to a larger document, at the point it's ingested into an application. If you're applying it to short strings, something's not right -- where did those strings come from?

Taking an ID3 tag example, if you are mass-converting/sanitizing/etc. tag titles and other similar metadata, the strings are often very short, sometimes only even a single codepoint or character, and proper assumptions of encoding can not be relied on because so many people violate specs and put whatever they want in there, which is the whole point of wanting to sanitize the info in the first place.

I don't think that's true. Looking at how it's encoded[0], it seems similar to many other country/language-specific encodings: bytes 0-127 are the control chars and latin alphabet and symbols, and is more-or-less ASCII, then 128-255 represent characters specific to the language at hand.

The only way you'd successfully decode Shift-JIS as UTF-8 is if it essentially is just latin-alphabet text (though the yen symbol would incorrectly display as a '\'). If it includes any non-trival amount of Japanese in it, it'll fail to decode as UTF-8.

As for whether or not you can then (after it fails to decode as UTF-8) use statistical analysis to reliably figure out that it's in fact Shift-JIS, and not something else, I can't speak to that.

[0] https://en.wikipedia.org/wiki/Shift_JIS#Shift_JIS_byte_map

I in fact can assume it. If the assumption is wrong then that's someone else's problem. 15 years ago I wrote a bunch of code using uchardet to detect encodings and it was a pretty useful feature at the time. In the last decade everything I've touched has required UTF-8 unless it's been interoperating with a specific legacy system which has some other fixed charset, and it's never been an issue.

There’s a difference between asssuming and not making a distinction.

Very few developers I’ve met know could make a distinction. They’d see a few off characters and think it’s some one-off bug but it’s because they’re both assuming an encoding.

Even if you said you’d pay them one billion dollars to fix it, they’d absolutely be unable to.

OK, for a billion dollars I will write a program that deals with KOI-8 or any other text encoding from the 1960s you care to name.

Please pay up front.

> Even if you said you’d pay them one billion dollars to fix it, they’d absolutely be unable to.

Unless you want it fixed immediately, then a million dollars should motivate almost any developer to spend a month learning, a month doing, and a few years on vacation. A billion is incomprehensible.

You can assume encoding is UTF-8, length is in meters, and timezone is UTC.

You just won't always be right.

The trick is to be wrong in precisely those situations in which you don't care.

Microsoft would like a word with you (utf-8-bom & utf-16)

Microsoft wants a word with everybody.... and that word is "pay us more money".

Universal UTF-8 is a hope we aspire to, not a reality we assume.

The way you get that reality is you do the opposite of the recommendation of Postel’s law: be very picky about what you consume and fail loudly if it’s not UTF-8

Reply on news.ycombinator.com