Comment by pak | Hacker News Reader

Comment by pak | Hacker News Reader

"This is SUCH a classic sloppy programmer mistake that I'm disappointed"

Oh come off of it. This happens everywhere on the web on probably something like 25% of websites. And it's NOT always the consuming program's fault: very often somebody upstream, e.g. the hosting company, the person that wrote the HTML, the source of an RSS feed being inserted into the page etc. etc. forgot to encode something the way somebody else expected, and you as the poor guy at the end of the chain gets a document with multiple encodings improperly embedded into it. Inevitably you have to make some bad decisions and not all corner cases are handled.

Somebody once reverse-engineered the state chart for how Internet Explorer handles documents with conflicting encoding declarations and I kid you not, it must have had >20 branches spanning a good few pages. Officially, the correct order of precedence is (http://www.w3.org/International/questions/qa-html-encoding-d...):

1. HTTP Content-Type header

2. byte-order mark (BOM)

3. XML declaration

4. meta element

5. link charset attribute

but that's not how every browser does it, because the W3C sort of declared that after things on the Real Internet (TM) had already gotten out of hand. I hate to resuscitate Joel posts but Unicode is not easy to implement right.

Twitter said that the message in question is UTF-8. The message recipient decoded the message with something that is not UTF-8.

That's a sloppy mistake and should without doubt have been caught in testing.

Yup, it can also be much more serious than some garbled text. Google "canonicalization vulnerabilities" and see stuff like this: http://www.amarjit.info/2009/09/canonicalization-vulnerabili...

How do you (or the OP) know the implementation details of the consumption path for display? The author assumes it was the direct JSON feed from Twitter. That's overly simplistic and betrays his inability to understand how these sort of things are put together. This is a public, real-time event; more likely than not tweets were filtered through one or two other components to separate the interesting material from the noise. You have no idea what those intermediary components were, who set them up, and if they were all decoding/reencoding correctly. Perhaps they were going through other Twitter users' clients, which could have contributed to the bug.

Edit: yes, there were definitely people curating the questions, so it could not have been as simple as pulling one JSON feed. http://www.theatlantic.com/politics/archive/2011/07/how-obam... https://twitter.com/#!/townhall/july-6-curators

> You have no idea what those intermediary components were, who set them up, and if they were all decoding/reencoding correctly.

...That's kind of the point. There was someone being sloppy there.

The company responsible seems to have responded in the comments:

"It was definitely a mistake on our part. The problem was not the encoding on our data feed, but the HTML document was sent with ISO-8859-1. The second we inserted the twitter text into the DOM, the browsers interpreted the UTF-8 string as ISO-8859-1. Our visualizations are hosted on other platforms, and in this case the server was not configured to send UTF-8 with text/html even though the HTML file was encoded as such. It was the only issue (albeit a pretty obvious one) during an otherwise flawless event. I apologize to President Obama, Speaker Boehner, and Jack Dorsey for the mistake. If the readers of the blog think it was stupid, imagine how we felt. dev environment != production environment. If we would have just included a <meta charset="utf-8"> in the HTML head, then this would not have occurred.

The big take away is don’t make assumptions about other platforms (especially when it comes to encoding), and always include charset meta tag." [emphasis mine]

Including <meta http-equiv="Content-Type" content="text/html; charset=utf-8"/> is like the first thing you do when starting front-end development.

Every time you don't validate...God kills a kitten. Please, think of the kittens.

This has been a Public Service Announcement. Please code responsibly: http://validator.w3.org/

Just for the record, the correct spelling is now:

  <meta charset="utf-8">

Was emailing involved? We've been getting so much support questions regarding encodings of HTML emails (we're a programmatic email service: http://mailgun.net) that we actually decided to become the first MTA to enforce utf-8 transcoding on the entire traffic: randomness in -> clean utf8 out and everyone is suddenly quiet&happy.

Something like this could be done as nginx/apache module: which detects encodings of the data and transcodes the HTML output into utf8 - could be useful for some cases.

Actually, the correct order is defined in the HTML5 spec now: http://dev.w3.org/html5/spec/parsing.html#determining-the-ch...

Really, it's not that hard. 99% of the job is done for you. Just use some UTF-8 encoded text to test your app before letting the president of the United States use it.

There is no excuse for not testing your app for basic i18n brokenness. It's 2011, not 1998.

Reply on news.ycombinator.com