Hacker News Reader

A strategy I've seen for dealing the inability of XML to handle overlapping tags, is to treat the tagging as an annotation layer on top of the node with the data:

  <doc>
  <data type="text">
    This is some sample text.
  </data>
  <annotations>
    <tag1 start="1" end="3" comment="foo"/>
    <tag2 start="2" end="4" type="bar" />
  </annotations>
  </doc>

The start and end are usually byte offsets from the start of the text content in the data node. It still sucks, but at least you could apply the same general stragegy to more than just text data - I've seen it used with audio/video where the offsets are treated as time offsets into the media.

Good idea. You would have to edit your annotation layer in case the text data changes though.

It seems klutzy and yet fully in the spirit of XML.

Now you've lost human editability/readability and could just as well encode that in a non-XML format.

Yep.

In my experience, the traditional solution for editing with these kinds of hacks is to write a buggy piece of shit custom GUI so people can edit documents. That way, the complaints shift away from your lousy data format to your lousy UI. Problem solved!

I would argue that the inline way of annotating things in XML is actually ok-ish if one absolutely needs human edit-ability, but otherwise bad design.

  {text: "Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua.",
   annotations: [{tag: "sometag", ranges: [{from: 12, to: 26}]},
                 {tag: "sometothertag", ranges: [{from: 21, to: 39}]}

Note that this also removes the limitation that annotations have to be consecutive.

crafty, but for your consideration: that places the burden upon every library author to be "accounting accurate" to any edits, and the only way anyone would know that it's not correct is to visually inspect the output text

also, as I get older I have a deeper and deeper appreciation that "offset" and "text" are words that are fraught with peril

Fair critique, but shouldn't strings be immutable anyways? Once you bring editing into play, you'll probably either want something like a rope or some CRDT and then you have more effective means of tracking positions, than manual offset computations, as part of the data-structure.

What about using inline "floating" checkpoints for the ranges, instead of character indexes?

  {text: "Lorem ipsum {{1}}dolor sit {{2}}amet{{3}}, consectetur{{4}} adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua.",
   annotations: [{tag: "sometag", ranges: [{from: 1, to: 3}]},
                 {tag: "sometothertag", ranges: [{from: 2, to: 4}]}

You are absolutely right that XML is better for document structures.

My current theory is that Yjs [0] is the new JSON+XML. It gives you both JSON and XML types in one nested structure, all with conflict free merging via incremental updates.

Also, you note the issue with XML and overlapping inline markup. Yjs has an answer for that with its text type, you can apply attributes (for styling or anything else) via arbatary ranges. They can overlap.

Obviously I'm being a little hypabolic suggesting it will replace JSON, the beauty of JSON is is simplicity, but for many systems building on Yjs or similar CRDT based serialisation systems is the future.

Maybe what we need is a YjsSchema...

https://github.com/yjs/yjs/

Yjs isn’t a document structure is it? It seems to be a library for collaborative editing, but I’m not seeing something suitable for marking up a document, or am I missing something obvious?

You can add metadata to ranges inside the text.

This is actually one of the things processing instructions are useful for - but you would need to define the data within the PI, since they don't have attributes.