Hacker News Reader

Comment by tialaramex | Hacker News Reader

The completely unacceptable thing about XML 1.0 is that it can't express all of ASCII.

You might be thinking, "Duh, just escape it". Nope, ASCII control codes are forbidden you can't escape them, the escaped form is still forbidden. You can escape say, a backslash, or a quote mark, or an angle bracket, but you can't escape say a vertical tab.

So if you want to move arbitrary ASCII text (or Unicode text, since ASCII is a subset of Unicode) with XML you need to Base64 encode it, or you won't succeed.

15-20 years ago I wrote code which needed to square this particular circle where clients insisted that they want XML text and yet they also demand to handle the forbidden values, my code would just convert all impermissible values to Unicode's U+FFFD the Replacement Character.

View on HN

That's a feature, not a bug.

Control codes aren't text. Heck, HTML doesn't even support tabs -- whitespace is just generic whitespace, interpreted as a regular space.

If you want to insert non-typographic elements that affect the layout/presentation, that's what <elements> are for.

While if you need to encode binary data that can represent ASCII text, then yes -- Base64 is an ideal solution. That guarantees it won't be garbled by other XML processors.

The W3 basically says all this here [1]. ("Use of control codes in HTML and XHTML is never appropriate, since these markup languages are for representing text, not data.")

[1] https://www.w3.org/International/questions/qa-controls

> HTML doesn't even support tabs

Sure it does. As does XML. You just have to choose deliberately to use it, via such means as <pre> or the CSS `white-space: pre`, or, in XML, the xml:space attribute.

Incidentally, HTML and the web is a place where the superiority of tab-based indentation to space-based is particularly apparent, because you’re far more likely to encounter very narrow viewports, where reducing the width of tabs is useful, while still keeping a reasonable indentation level on wide enough screens. I do something similar to this, for example:

  :root {
      tab-size: 4;

      @media (max-width: 50rem) {
          tab-size: 2;
      }
  }

Exactly. If you do want to encode control characters, the way to do that would be as character entities (which are simply text like &) and handle the translation of these entities at the application level. As an example, if we want a newline to be removed for presentation (or translated into a newline symbol like in Microsoft Word), the right place to do that would be at the rendering time and not in the data representation layer (which is where XML operates).

You could also use tags, for example <br> tag in HTML.

After all it is Extensible Markup Language and the functionality for adding custom tags is the basic feature.

And as suggested above, base64 encoding and application level decoding is also an accepted practice.

All these techniques point to handling the rendering at the application level.

> base64 encoding and application level decoding is also an accepted practice

I feel seen. I remember suggesting this during a design discussion where the issue was how to embed an arbitrary document in another, where the embedded document could even be ill-formed. Everybody told me it was a gross and dirty hack and ignored the suggestion.

It is quite a common pragmatic architectural pattern in the HTML architecture. For example, data urls where a MIME type representation is encoded in the URL format is used for embedding images and other MIME types.

So, if one is building an application and needs to render a proprietary data object, this is a good way (not the only one) to handle it using a client library.

> Use of control codes in HTML and XHTML is never appropriate, since these markup languages are for representing text, not data

Is XML meant to represent only text and not data?

XML 1.1 doesn’t support all of ASCII either. https://www.w3.org/TR/xml11/#NT-Char:

> Char ::= [#x1-#xD7FF] | [#xE000-#xFFFD] | [#x10000-#x10FFFF] /* any Unicode character, excluding the surrogate blocks, FFFE, and FFFF. */

I’m puzzled at the total non-mention of U+0000 in the document, which is a valid Unicode and ASCII character but is not supported by this definition of Char. So it seems like it’s actually (and using better terminology) “any Unicode scalar value, except U+0000, U+FFFE and U+FFFF”.

The same restriction you noted on entity encoding still holds: https://www.w3.org/TR/xml11/#NT-CharRef

The exclusion of NULL is almost certainly for C style string compatibility

Reply on news.ycombinator.com