Hacker News Reader

Comment by stephen_g | Hacker News Reader

It's good to know that it's on the increase. Still, character encoding seems to be something that is understood by very few people... I wonder how many of these sites are just reporting UTF-8 or something because their web server defaults to it, and not actually encoding special characters properly?

It's certainly very easy to do - I had the problem a little while back setting up a little web app where the web server and MySQL database were both UTF-8, but the db connection was defaulting to ISO-8859-1 or something, causing all sorts of issues with curly quotes etc.

View on HN

> I wonder how many of these sites are just reporting UTF-8 or something because their web server defaults to it, and not actually encoding special characters properly?

I think they are measuring the encoding that is picked before adding a page to the index. Which would be a combination of explicit information (headers, xml and html metadata) and heuristics (which may fail, but are useful when the explicit information is missing or — even though ignoring explicit metadata is bad — obviously incorrect).

It also seems they are looking for a subset encoding once they have the metadata; the posts describe explicitly labelling ASCII when the contents are within that subset.

But their methodology has changed from the previous two posts: latin above ascii in 2008 didn't exist in http://googleblog.blogspot.com/2010/01/unicode-nearing-50-of... or http://googleblog.blogspot.com/2008/05/moving-to-unicode-51.... .

Yes I wondered this. I would love to see figures for incorrectly labelled on that chart.

The great thing is that one now only has to explain to people to use UTF-8 in all cases, which is preferable to having to fool around with a lot of language-specific encodings.

And it helps that almost all editors, these days, also default to UTF-8 encoding. So you can just copy/paste special characters and it works...

Java is among the last holdouts for Latin-1-encoding - you still have to jump through hoops with Spring MVC to properly send UTF-8 encoded responses

http://stackoverflow.com/questions/3616359

Also the encoding for .properties files is specified as Latin-1

http://en.wikipedia.org/wiki/.properties

Mysql's character encoding is pretty hilarious. I love how its "latin1" is nothing of the sort (it's actually cp1252 with eight characters shuffled around).

All browsers use UTF by default and all web pages should be serving UTF-8: http://hsivonen.iki.fi/accept-charset/

Reply on news.ycombinator.com