The Goals of XML at 25: and the one change that XML now needs
schematron.com
schematron.com
I also find TOML to be much a cleaner syntax than either of the two, but maybe we should just give up and make a JSON-C (for Configuration, I just made this) which supports comments.
One can usually answer the question "am I going to frequently have deeply-nested tables and/or long strings for keys" in advance, and if the answer is no then TOML is practically ideal. If it proves to be wrong occasionally than JSON is a reasonable fallback, as is just coping with the verbosity of TOML.
This isn't a snipe at YAML specifically. It's a repeated pattern I've seen with many technologies. There's a certain mode of advocacy you can use to invoke this on a certain set of the programmer population, and it works reasonably often.
(The "better than the X you all hate!" part may be obligatory; I haven't studied this in a careful academic sense, I've just observed this pattern a few times. Leveraging an emotional reaction against some tech and using it to suppress a rational examination of the proposed replacement on its own terms may be the mechanism of how this is done.)
<?xml version="1.0" encoding="UTF-8" ?>
<!DOCTYPE greeting [
<!ELEMENT greeting (#PCDATA)>
]>
<greeting>Hello, world!</greeting>
# Ranking of 1998 home runs
---
- Mark McGwire
- Sammy Sosa
- Ken Griffey
# Team ranking
---
- Chicago Cubs
- St Louis Cardinals
It’s clearly not perfect but IMO it’s the best right now.
Because we want a low-noise human configuration language.
XML and JSON certainly aren't it.
TOML and HOCON are low-noise, but share many of YAML's drawbacks.
The "O" in TOML is not deserved in my opinion.
Just like YAML, there exist a sound subset of XML that is quite usable. Just like YAML, XML is hard to write correctly, but I would say the XML errors are much easier to spot.
Both JSON and YAML have the problem of lacking tagged unions. Also, the XML data definition can be either extra complexity or a great tool depending on your usage. (But DTDs use an atrocious language either way.) None of those 3 standards have a really universal type representation, and while there are substandards for XML that extend the types, I've never seen anything that could encode even the C type system.
As the OP said, the quest for the perfect data language is still on.
Poor Norway.
https://hitchdev.com/strictyaml/why/implicit-typing-removed/
So do I. XML is sane and has a lot of tooling readily available.
YAML, with its significant white space, is one of the shittiest formats that I have ever encountered.
comments. trailing commas
https://nigeltao.github.io/blog/2021/json-with-commas-commen...
What are the types in the following YAML document?
a: true
b: on
c: yes
d: TrUE
e: aye
Having spent a lot of time in XSL, I think its core feature is the matching-and-synchronization story. Essentially, every XSL template can be cut up into a sequence of CPS blocks that perform a mapping operation on the sub-calls to apply-templates.
Porting the matching & template apply syntax to Python is fairly easy — the ergonomics are great! The problem is that mapping operation is functionally impossible for me to implement, efficiently, in Python with parallelism. The result is that even though the big implementations of XSL are 100-1000x slower than Python in terms of serial computation, they handily beat Python in deeply concurrent constructs.
My understanding of `asyncio` is insufficient for me to determine if it's even possible to map `asyncio` onto multiprocess pools and get back the data at the yield point.
And — even if it were possible — the batteries-included version of ElementTree reloads & reparses the entire document in every subprocesses, completely destroying the performance.
"XSLT is a failure wrapped in pain. There’s no job for which XSLT is the right tool. If you think you found a job for which XSLT is a good tool, chances are the job itself is fucked up" – masklinn
I think this had a lot of unintended consequences that has hurt XML in the long-run but has nothing to do with XML itself.
I wonder, for what are the use cases today one would prefer XML over JSON/GRPC/GraphQL etc ?
https://annevankesteren.nl/2007/10/xml5
I wrote a parser for it, but it never got traction.
I might have to rewrite it again.
http://www.textuality.com/xml/xmlSW.html
XML-SW drops the internal DTD subset entirely, including entity declarations; this means it doesn't support named entity references except for the predefined ones (apos, quot, gt, lt, amp). I think DTDs are one of the ugliest things in XML (and also a well-known source of security vulnerabilities), and it is great that XML-SW (unlike XML5) mostly kills them.
I'm also not sold on the whole "HTML-style error-recovery" idea in XML5. In my mind, the syntactic strictness of XML is a virtue not a vice.
I'd support killing processing instructions as well. They are rarely used, and XML namespaces could be used as a replacement.
Having used and written a parser for a similar recoverable localization language (https://projectfluent.org/) I'm sold on it.
It makes a lot of things easier. It's kinda like adding trailing comma to lists. It's both boon when writing lists by hand and generating it via code.
> I think DTDs are one of the ugliest things in XML (and also a well-known source of security vulnerabilities), and it is great that XML-SW (unlike XML5) mostly kills them.
Hm, that's strange. I'm pretty sure XML5 killed DTD outright, other than having a doctype definition. Perhaps the spec was outdated a bit.
XML5 had a few big ideas:
- Ending DTD
- Unquoted XML attribs
<elem atr1=value>
<a><b></a></b> // Became <a><b></a> <b/>I'm just going by what is at https://annevankesteren.nl/2007/10/xml5 which gives the impression that <!ENTITY> declarations are still allowed.
I think there might be a misconception here. HTML5 does have some ad-hoc rules around script, style, and noscript elements, but the vast majority of tag inference it does is straight from SGML, from which both HTML and XML are derived. Tag inference in SGML is driven by (parent) element content models and per-element tag omission indicators. For example, a minimal HTML doc can look as simple as
<title>XYZ</title>
<p>body text
<html>
<head>
<title>XYZ</title>
</head>
<body>
<p>body text</p>
</body>
</html>
SGML is formalized in ISO 8879:1986, and there's absolutely nothing vague or speculative about the kind of tag inference it does. It's just that most people don't seem to know SGML and come up with ad-hoc XML extensions that can't be taken seriously without reference and alignment within the greater SGML context when XML itself is specified as a proper SGML subset.
It's interesting that you'd be willing to trade processing instructions for XML namespaces. Most experts, including the article's author, seem to agree that namespaces and XML Schema are the worst XML offenders, taking the just gained simplicity of XML away on the spot.
XML Schemas were the End of its simplicity.
XML is awesome for configurations: with proper XML Schema and proper editor (I use Jetbrains Idea, but there are other options) you'll get almost full validation, auto-complete for free.
I would prefer XML for any markup.
JSON have one important advantage over XML without DTD/Schema: it's impossible to express array in XML. Another less important advantage is that there's only one way to stringify graph of arrays/maps into JSON. With XML there's no "official" standard, there are plenty of different ways to write maps, for example. But DTD/Schema solves those shortcomings.
The advantage that a ML can have over another is how easy it is to:
- parse by the machine
- read and edit by humans
- esoteric features like cyclic dependencies
- sizeI had to edit Liquibase configs in YAML and XML. YAML was a fucking nightmare.
I still get flashbacks to maintaining XSLT sometimes. It was a tool that didnt really need to exist and arguably did more harm than good by being "recommended".
I remember very clearly that although XML had tooling for schema validation it was awful and people tended not to get beyond the "DTD vs XSD" debate before giving up on it. Both of them sucked.
Then there's XPath. Why XPath? Because if your markup language doesnt map to normal programming data structures you need a band aid to fix it.
As for SOAP.... that can die in the fiery pits of hell with XMLSpy.
DTD is mostly just an attack vector. XSD or Relax NG are both solid. XQuery was kinda useful but XML people went too far and started adding shit that it doesn't belong to it.
XSLT like has one or two credible use cases, like replacing entity expansion from DTD. Rest of it was just like XML database doomed from the start.
SOAP. I never saw it actually being used.
Most of this tooling has equivalents that are barely used because they didnt really need to exist in the first place (e.g. XSLT) or were a band aid over problems with XML itself (XPath / XMLSpy).
That leaves JSON schema, which - still more pleasant than XSD or DTD.
How do you express easily “every element that has X in its path” with a programming language? Of course you can write such a program but it is similar to regex - you could write a FSM but it is much easier to use a DSL for that.
Not a fan of XML, but I think XML Schema is one of the better Schema solutions I have used. If you doing something with XML and not take advantage of XML Schema it seems like a waste.
XML for markup is okay, but usually I just use a subset of HTML5 when doing that, XML would have the benefit of having XML Schema support for validation, but I have class that can do that for me with HTML5.
- numbers are a mess. You can have the full decimal numbers, the "e" notation, and sometimes you have to use strings for number that are too big for JavaScript.
- Map keys order is not significant, so it can be any order.
Also: why the fq order matters
So, whatever reason JSON won it wasnt that.
That said, xml could have worked out fine if there had been sane management of it's evolution. But now the cat's out of the bag, and we got json, and the "named-to-be-a-loser" YAML.
Maybe someday that kind of ambition will lead to something great, but we needed a step back.
The post here for your convenience:
"What is XML good at?" (For the purposes of this post, I'm including HTML in the XML family.)
XML/HTML is good when:
1. You have two dimensions of markup you want to do. That is, you have a clear distinction between what is a new "tag" and what is an attribute on that tag. If you can't almost instantly decide whether some feature you want to add works as an attribute or a tag, you probably shouldn't be in XML.
2. Almost every tag one way or another contains some text, the third dimension that XML supports. A proliferation of tags that never contain any text is a bad sign. A handful may not be a problem, e.g. "hr" in HTML, but they should be the exception.
3. You have a really good use case for XML namespacing, the fourth dimension of information that XML supports, in which case there's almost no competition for a well-standardized format, as long as you're also using the previous three dimensions.
There's sort of this popular myth that XML is useless, which I think isn't because it's true or that XML is bad, I think it's because in general, most times you want to dump out a data structure #1 isn't true, let alone #2 or #3. In a lot of data sets, you've only got the two dimensions of "simple structure" and "text", not annotations on the structure itself. (Or, perhaps even more accurately, they end up implicit in the format itself, and the format is constant enough for that to be just fine.) A lot of stuff in the 1990s and 200xs used XML "because XML" even though it clearly failed #1. XML is really klunky when you don't want that second dimension because the XML APIs generally can't let you ignore it, or they wouldn't actually be XML APIs.
On the other hand, when you learn this distinction, you do come across the occasional JSON-based format that clearly really ought to be XML instead. You can embed anything you want into JSON, but when you're manually embedding a second structure dimension into your JSON document, it loses its advantages over XML fast. If you've ever seen any of the various attempts to fully embed HTML into JSON, without leaving any features behind, you can begin to see why XML or XML-esque standards like HTML aren't a bad idea. HTML is much easier to read for humans than HTML-in-JSON-with-no-compromises.
And if you've truly got the four-dimensional use case, XML is really quite nice. When you need all the features, suddenly the libraries, completely standardized serialization, and XPath support and such are all actually convenient and surprisingly easy to use, for what you're getting.
Some examples: HTML is a generally good idea. SVG is a middling idea; it passes #1 and #3 but fails #2. SOAP and XML-RPC is generally a bad idea; SOAP fails #1 and #2 but sort of uses #3 and XML-RPC fails all three. XMPP I actually think is pretty solid as an XML format (mere network verbosity problems can be solved with an alternate encoding, though admittedly that becomes non-standard), and in a lot of ways, the real problem with XMPP isn't so much the format itself as that people are not used to dealing with the four-dimensional data structures that result. People expecting IRC-esque flat text are not expecting such detail. Using the fourth dimension of namespaces for extensibility is neat, but few developers understand it, or want to.
How about: markup languages are good for encoding text, and everything that isn't directly part of text content ("metadata") goes into attributes? Arrives at about the same judgement regarding the relative merit for the use cases you mentioned.
I'd like to add that, while XMPP is XML-based, it might be more suitable to use a full-blown SGML parser to deal with XMPP streams being presented as incomplete XML (eg end-element tags missing) which SGML can deal with in a formal way (end-element inference).
It's not that end tags are missing in XMPP, just that the "document" is streamed so it isn't all available at once. Any common SAX-style parser can deal with streaming documents.
- It must be easy to use. The harder it is to set up, the less people will want to use it. The harder it is to work with, the higher the maintenance costs and the greater the number of mistakes and security holes. Poorly defined specs and a cavalier attitude towards the users of these technologies have trained us to just accept bad quality and make do.
- It must be efficient. Text formats are convenient, but wasteful of both CPU and bandwidth. Going forward we need to think more about our energy impact. There needs to be a way to have our cake and eat it too!
- It must be worry-free. Codec differences are a security risk. Bad defaults are a security risk. Lack of forward compatibility is a security risk. Today's hostile world does not at all resemble the world in which our current formats were developed.
This is why I'm developing https://concise-encoding.org/
The date format is not ISO8601 because ISO8601 has too much cruft, bad defaults, and is missing real timezone data. I'd have been fine with the cruft and bad defaults, but lack of timezones is a deal breaker.
Must it? Are text-format messages really that much of an impact on traffic levels or CPU use?
Human readability is a very important, probably more so. ASN.1 is pretty efficient, but I wouldn't wish it on my worst enemy.
[1]: http://lists.xml.org/archives/xml-dev/202107/threads.html
When XML was cool I was subscribing the list and followed (or tried to follow) the discussion. It was my first exposure to the Web standard world. The archive suggests that the list is mostly dormant, but still.
I classify them as sane XML ideas and some as complete utter garbage.
Good parts
- Schemas for documents - If you have a single schema file, and a single XML to do validation its KINDA nice. If you limit yourself to a basic set of XSD types its okay ish. - The general idea have XSD's express document structure and have a standard for it is kinda nice.
Bad parts
- people go absolutely bonkers with schemas, have schemas in schemas, do complicated stuff that can hardly be expressed in modern type systems, and even harder to generate code for. Things like PEPPOL or UBL are extreme horrible overly complex solutions, and you need an army of devs that have the patience to implement it properly.
As a result, everybody does the bare minimum and gets their own flavour and you're still making per supplier code since everyone just tries to make it work. Our code base has a ton of 'fixers' per supplier to convert their iffy files to something sane. We started by asking them to fix their files but in the end it was just waaaaaay faster if we dealt with it, because the complexity of truly understanding XSD's is quite complicated.
- The a part of the standards community, feels more like schema designers than actually implementers. It feels like whenever a committee designs a standard they're like "WE'RE DONE!" without actually implementing it themselves across languages and see how hard actually is to generate something valid. These organisations are head-acheinducing money sinks where there bad schema ideas trickle down.
- It can basically do too much, what you want is a tiny expression of your types.
- JSON Interopability: You can't just convert JSON to XML without schema knowledge. In XML this can `<user><name></name></user>` can either be an item in an array [], or a hash. You need to know the schema to understand how to deserialise or generate properly, you can't cheat.
- Its super verbose, and hard to read. When you're staring at 200 lines of XML, you're brain has to parse all the verbose namespaces and what not in order to pick out the data.
I also deal with +1GB XML files and then the tooling to support large files grows thin. And you resolve to SAX parsing and emitting nodes. It aint fun. In the end I had to write a lot of tools myself to make it manageable to work with many documents.
Anyway thanks for reading this far, I really needed to vent hahaha
Namespaces are high on my list of decent IDs with botched implementations killing the concept. The thing which made working with them so painful was that the tools pushed all of their work off to the user and did inconsistently.
It's not just that you have to know they exist but you have to know that, for example, you're going to need to pass namespace declarations to many APIs because the implementers chose to require those as parameters rather than reading the declaration out of the document being parsed and depending on which tools you have the misfortune of using you might have a source like “myns:foo” but the parser will require you to use "{http://example.org/myns}foo" for queries. Similarly, the default could be inherited so you could simply write '<myns:foo bar="baaz"><quux /></myns:foo>' and assume that everything unqualified is in “myns”.
Imagine how much better this would have been if, a couple of decades ago, anyone had cared about the developer experience enough to fix the common tools. Non-broken namespacing, good error messages, easy to use validators and formatters, etc. would have removed so much constant friction which lead everyone to run for the door as soon as JSON started to get momentum.
> - The a part of the standards community, feels more like schema designers than actually implementers. It feels like whenever a committee designs a standard they're like "WE'RE DONE!" without actually implementing it themselves across languages and see how hard actually is to generate something valid. These organisations are head-acheinducing money sinks where there bad schema ideas trickle down.
Imagine if 10% of the money which went into barely-used standards development had gone into maintaining libxml2 (and maybe xerces). Basically any bit of work which went into XPath or related specs after 1999 was a write-off because those later versions effectively never shipped for anyone outside of exclusive .NET users.
XML now needs a database.
The fact it's built around schemas is great, and this makes it way better at representing serialized structured data on a file than JSON and anything schemaless. On the other hand though, it has perhaps one of the worst syntaxes known to man. I dread reading and writing complex stuff in XML (Plists are one of the worst offenders), as it is often true that a lack of thought at time of schema definition can causes terrible readability problems that tend to linger for years.
People are often in a hurry when releasing a new piece of software, and XML makes very easy for a hasty developer to overcomplicate things.
Certain pieces of XML basically require you to edit them through the GUI tool that created them, and the punishment for not doing so is having to recover a completely messed up XML file. I am personally a strong believer in the idea that a text file MUST be easy to edit by hand, because someone will always try opening it up in Vim and edit it that way.
If a given file format forces you to use (de jure or de facto) a specific program to edit it safely, it should be using a binary representation instead. Binary files are often smaller, and have the unintended additional bonus of making the file not editable in a text editor.
E.g. one of area where XML shines are user interface notations. A screen may have a tab that has another tab that may show yet another screen; a menu may have a submenu, and (although not recommended) a sub-submenu and so on. A field or text may appear here and there. The two chief things here are ordering and composition. XML is very good at that. Not as good as a dedicated UI language, perhaps, but, being generic, it's much more pliable an easier to tinker with.
...
Yeah, that checks out.
Char ::= #x9 | #xA | #xD | [#x20-#xD7FF] | [#xE000-#xFFFD] | [#x10000-#x10FFFF]
There's no good reason to do this (even null character, although C will not like it).
XHTML was a failure, rejected and ditched.
So if it lost data interchange and layout/interface specification... what is XML's relevancy (not counting legacy interaction/systems)?
The big thing I'd be curious about is what happens when new features are added: do they get added to the XHTML parsers as well or does that become a legacy path drifting behind the main web platform? I believe there were already some issues with Chrome along those lines when they added JS modules, for example.
Also, most browsers today no longer support XHTML 1.x "Strict" modes, the errors they surfaced, and instead just laxly parse all XHTML 1.x with the same parsers they use for "HTML5". (IE11 I believe was the last browser with XHTML "Strict" support and used a proper XML parser and error reporting when parsing XHTML.)
[1] Outside of how XHTML 2.0 influenced HTML5's list of "semantic DIVs" the tags like ARTICLE and SECTION that don't have any meaningful semantics in current browsers but are nice when you get bored of surrounding everything in DIV. XHTML 2.0's standard attached actual meanings to those tags and had a lot of specified behavior for how those tags would operate. The only surviving "tag behavior" from XHTML 2.0's plan was DETAILS, I believe. I also believe the exact lists of tags are disjoint and XHTML 2.0 had tags that didn't even make into the official "semantic DIV" bucket, are out of spec in HTML5, but HTML5 browsers will mostly just ignore and fallback to treating unknown tags as fallback DIVs anyway.
And fixed: https://vbezhenar.com/test-valid.xhtml
It works in Chrome for me, parsing failed as expected on the invalid file. Also you can't use document.write from JS, so DOM has special treatment for XHTML as well.
Google, who cares deeply about their search moat, did not want easily parsable documents that revealed the wealth of underlying data to machines. It might have made it too easy to mine documents and present search results that answered common factual queries.
Also, by forging ahead with landing nonstandard feature after feature in Chrome, they were able to bleed away Firefox market share and bend the web to their will.
The only mistake XHTML made was the mime type requirement.
For this to happen, a webpage has to be served with an XML Content-Type by the server—specifically, application/xhtml+xml. HTML documents are served with a Content-Type of text/html. If you write a page entirely in valid XHTML markup, but serve it with an HTML media type, the browser treats it not as XML, but as invalid HTML, like any other tag soup, and turn it into something approximating what you meant.
However, some browsers couldn’t handle application/xhtml+xml, and would simply prompt a download dialog for the page rather than displaying it. Well, one browser did that. You know the one… Internet Explorer. Because support for the new media type wasn’t widespread, the W3C allowed serving XHTML 1.0 pages as text/html as a transitional mechanism. They wouldn’t get any of the “benefits” of XML parsing (like the YSOD), but they’d display in IE. XHTML 1.1 and the stillborn 2.0 were required to be served as application/xhtml+xml, and thus saw only the barest minimum of adoption. As transitional mechanisms are wont to do, XHTML 1.0‐as‐text/html stuck around, W3C’s reputation took a blow, and WHATWG seized the reins, codifying browser practices into what’s now known as HTML5.
Incidentally, this is the origin of the space everybody puts within a self‐closing tag. An XML parser would see no difference between “<br/>” and “<br />”, but some tag soup HTML parsers of the time did. The XHTML spec had an appendix dedicated to these compatibility tricks that were solely needed because people were serving pages with the only Content-Type that IE understood. https://www.w3.org/TR/xhtml1/#guidelines
Before that it was more "undefined" behavior than "illegal" as older HTML specs simply didn't consider it.
Why not? AFAIK it worked well. See SAP and many other products.
I'd say the immediate thing that comes to mind is that there isn't a one-line-code that would deserialize an XML message to language-universal collections (arrays/lists/vectors, dictionaries/hashmaps/assoc. arrays). The closest was XStream in java-land, which I used heavily before JSON became widespread.
Secondly, it doesn't have a natural specifier for THIS IS A LIST. You're guessing or relying on mappers/marshalling to produce such things, while JSON can naturally map from JSON to an object graph without excessive mappers, annotations, etc.
Other complaints: The attribute/tag dual-duty was also problematic, confused the entire problem space.
The complexity of XML features (namespaces and bunch of other standards) is hidden in libraries. And some of those features might be useful, only to be re-implemented in JSON, with same amount of additional complexity in libraries.
Just recently I used lxml in Python to parse RSS and it gave me collection like if I used json module. Moreover, I used XPath extensively to pick parts of the XML I was interested in. This would have been more complex with manual traversal. Of course there are Python bindings to jq, but in general the jq language isn't as widespread as XPath. Then there's the slight annoyance of accessing attributes via dedicated getter methods as they're "special" as you mention. But in all, I haven't noticed millions of lines of extra code and complexity.