Hacker News Reader

Sorry if this sounds a little clueless, but it sounds like the problem is because there's multiple Unicode standards (UTF-8, UTF-16, UTF-32). So it seems like if you just re-encode everything into one of these before committing to the storage layer, that you'd avoid this problem altogether, and you'd be able to do operations in the storage layer correctly too.

View on HN

A Unicode string is an array of Unicode's building blocks: Code points. Normalization/composition refers to how we use code points to represent characters. Encoding refers to how we use bytes to represent code points. The two concepts exist at two independent levels of abstraction.

Accurate but so arrogant. HN purity achieved.

Sorry, I didn't intend that. I honestly considered it a reasonable question.

No worries, your reply was a concise clarification of the issue and terminology, and a reasonable answer; I didn't perceive it as arrogant in the least.

OP here; I didn't perceive it as arrogant either. I think the "arrogant" accuser might have a point about some stuff I see on HN, but this reply to my post just isn't it.

Yeah, no. That is actually a different issue. Curious if you read the OP?

But, no, you do not avoid issues with unicode normalization by ensuring everything is in (eg) UTF-8. Unicode normalization is a thing within UTF-8.

Reply on news.ycombinator.com