Hacker News Reader

Comment by calvinmorrison | Hacker News Reader

I always wonder why we don't use this.

Back when rolling your own application level protocol on top of TCP was common (as opposed to using http, zeromq, etc) I frequently used file/record/group/unit separators for delimiters, and considered them an underrated gem, especially for plain-text data where they were prohibited to occur in the message body so you didn't have to escape them (still good to scan and reject messages containing them). As a modern example they (and most other ASCII control characters) are disallowed in json strings.

You can put control characters in JSON strings, you just need to escape them.

The way I read the json standard, the only way to include control characters is to encode them as hex. For example BEL can be encoded as "\u0007", but escaping it by using a backslash followed by a literal BEL character is not allowed. So literal control characters should never be in json text.

This makes me wonder if the Escape control character (\u001B) would work in a JSON string. Time to go test things out. :)

MLLP (Minimal Lower Layer Protocol) -- used extensively to transmit HL7 in health systems -- uses file separators to delimit messages.

  OB vertical tab

  <content>

  1C file separator

  0D carriage return

I wrote one of the most popular translators for MLLP, which converts it to HTTP [1].

---

P.S. Ironically, HL7 messages have something literally called a "field separator" but don't use the field separator character, usually they use vertical bar.

[1] https://github.com/rivethealth/mllp-http

CSV is the javascript of the tabular data world.

Everyone thinks they can do better, but nothing's more widely supported (for a sufficiently generous definition of 'supported')

Unfortunately CSVs vary a lot in the wild. Some people use commas as a delimiter, some use semi-colons. Escaping rules vary. And the text encoding is not specified.

I randomly generated some CSVs and fed them into Excel and Numbers and they were differently interpreted.

This is why I tend to use the Pg COPY version of TSV - works beautifully with 'cut' and friends, loads trivially into most databases, and the 'vary a lot' problem is (ish) avoided by specifying COPY escaping which is clearly documented and something people often already recognise.

Generally my only interaction with CSV itself is to fling it through https://p3rl.org/Text::CSV since that seems to be able to get a pretty decent parse of every sort of CSV I've yet had to deal with in the wild.

Countries that use . as the thousands separator (e.g. 1.000) use , as the CSV separator.

Countries that use , as the thousands separator (e.g. 1,000) use ; as the CSV separator.

Why? Because that’s how Excel does it.

Errata: as the decimal separator, not as the thousands separator.

Funny thing, excel, which is the most common spreadsheet editor, does not practically support CSV files if you happen to live in countries where the default official convention is using commas for decimal points in numbers. Unless you go around and manually set stuff in how it imports it or you change your default settings. It has reached meme levels at my work.

Tab separated files are much better imo in not getting confused with the delimiter for a sufficiently sane tsv file.

Yes it does, but then it uses ; as a separator.

In a POSIX shell, I actually prefer to use the bell character for IFS.

  while IFS="$(printf \\a)" read -r field1 field2...
  do ...
  done

This works just as well as anything outside the range of printing characters.

> I actually prefer to use the bell character for IFS

Heaven help you if you cat the source file in a shell, though!

All downsides, no upsides.

You cannot edit it in regular editor, like csv/tsv/jsonlines.

There is no schema or efficient storage, like binary formats.

There is no wide library support.

Not all data is representable.

1 Editors can be improved 2 Same as CSV etc then 3 Libraries can be improved 4 Escaping characters exists

ASCII 1963 had 8 separators, 1965 reduced it to 4, and named them. See 6.3.12 of https://dl.acm.org/doi/pdf/10.1145/363831.363839

The task here is to explain why one should use this over CSV. By your own admissions, there is no reason to prefer this over CSV.

There's no standard for CSV files, thus no one can parse them properly

The only time you need to escape a character is if it's a control character that's rarely used, unlike the " and , characters

> There's no standard for CSV files

Literally untrue. (And were it true, it still wouldn't be a reason why one should use this over CSV—not sure what's so hard to grasp about the conversational/contextual premise here.)

> You cannot edit it in regular editor, like csv/tsv/jsonlines.

If only there were shortcuts on modern operating systems to allow us to do things that aren't readily on our keyboards. Like upper case characters. Or copy and paste. Or close windows. Our lives would be so much better.

If ASV had caught on, there could be common shared shortcuts to type them, and fonts would regularly display them (just like the unicode characters proposed). But CSV was simple enough and readily type-able.

> There is no schema or efficient storage, like binary formats.

I'm not quite certain where you're trying to go with this. Binary formats aren't really meant to be human readable in an average text editor. It doesn't know to differentiate 1, 2, 4, or 8 bytes as an integer or a float. Even current hex editors to make it easier to navigate these formats don't really know unless you are able to tell it somehow.

> There is no wide library support.

It's a critical mass problem. Not enough people are using them, so no libraries are being made.

> Not all data is representable.

I'm not quite certain what data couldn't be represented. f you can represent your data in CSV, you can represent it in ASV. It's all plain text that gets interpreted based on what you need. They're nearly a 1:1 replacement. Commas get replaced by unit separators, new lines get replaced by group separators. Then you have record and file separators to do with for further levels of abstraction if you need.

Re editors: The problem with USV is not that it's hard to type the characters, but rather than the newlines are completely optional. Which means that in general case, most line-based tools are not going to work with USV.

Now, the readme actually has that optional newline separator thing, but the optionality of it makes it completely useless, it seems like an after-thought. Fr example the first "real" USV writer I found, the "csv-to-usv", does not put them [0] and thus makes uneditable files.

And if we are going to end up with uneditable files, might as well go with something schema-full, like parquet or avro. You are going to have the same "critical mass problem", but at least the tooling is much better and you have neat features like schemas.

[0] https://github.com/SixArm/csv-to-usv-rust-crate/blob/30a0324...

Good catch! I just fixed csv-to-usv so it prints newlines now. You're right, Parquet and Avro are both great, for use with schemas.

> I'm not quite certain what data couldn't be represented.

What do you do if you receive data already containing a unit separator, or a group separator, and you need to put it into a field? The whole value proposition of ASV over, say, TSV is that you should never need to escape anything, but that's only possible by rejecting some input data.

I've sketched out a replacement for JSON which would use these characters - https://shkspr.mobi/blog/2017/03/kyli-because-it-is-superior...

CSV is honestly not that problematic. Figuring out if an field contains and comma and then properly quoting it is trivial. And fields without commas don't need quoting. Sometimes your application even guarantees no commas, especially if CSV is into it from the beginning.

I'm guessing you haven't worked in custom support where people send you their "CSV" files. Even the field delimiter varies (many Europeans use semi-colons).

No, I have. I don't consider abuse of the format a problem with the format. Though I can see how having to delimit with special characters will help the type of person who writes print(','.join(stuff)).

>I don't consider abuse of the format a problem with the format.

That's a fair point. But you could argue that when the abuse is so widespread, it becomes a defacto part of the format (even if it isn't in the RFC).

Not easily readable / editable using a regular text editor.

Do those characters map to something visually useful in (typical) unicode fonts?

That would be neat :)

Edit: Apparently, kinda (e.g. https://www.compart.com/en/unicode/U+241E )

Not the most creative....

According to their GitHub README:

```USV works with many kinds of editors. Any editor that can render the USV characters will work. We use vi, emacs, Coda, Notepad++, TextMate, Sublime, VS Code, etc.```

I loaded an example in my fairly generic Emacs and it worked out of the box. The separators were pretty small so I had to increase my font size to distinguish US from RS. And of course I have no idea how to enter those characters. I'm sure there is, but cut & paste worked.

Who here uses a "regular" text editor, let's be real

I'm fascinated that a lot of posters in this thread are not understanding the ideas and experiences, that the inventors of this file format had or made. They invented this format because it works for machines as well as for humans. Text editors can handle the proposed UTF characters just fine. Humans can see them. The only challenge is that it is cumbersome to type the delimiters. And that the format is not used in any relevant software (like Excel). Both are reason enough, that USV will not be used anywhere. But I can see why they went this way on their file format.

I don't really see what benefit it provides over CSV other than needing to escape less frequently. That hardly seems like it's worth it.

We might be able to see them, but for me they're just a blur unless I zoom in significantly, so I'll need editor accommodations just as much for these characters as if they used the already existing RS/FS/US/GS characters.

It feels like instead of fixing it properly, they went with an option that will still need tool improvements, will be controversial, and adds unnecessary details (e.g. the SYN they've added will be an active nuisance and I'd be willing to bet will get ignored by enough tools to become a hazard to data integrity).

I quite like an initiative to make use of proper record and unit separators, but this feels poorly thought through in several respects (e.g. their quirky escape characters that adds differently depending on the class of the following character will be a 'fun' source of bugs; that splitting records on LF requires three characters almost certainly will mean a number of tools will incorrectly treat those three characters as a unit, etc. -- these assumptions are based on how slapdash a lot of CSV parsing and generation is; if you want to compete with CSV you ought to learn those lessons)

CSV works for machines as well as humans, why do you assume or imply otherwise? Making the separator hard to type makes this ‘invention’ hard for humans to use. Using the glyphs instead of the semantic Unicode separators might also make this harder to use, even if you can understand why they did it, and to some degree it subverts the intent of the Unicode standard’s separator and glyph characters.

We don't need a new format which works for machines as well as for humans, because there are are tons of existing ones. You have CSV or TSV for wide support; JSONlines if you want very easy edit-ability and structure; and if those don't work for some reason, pretty much any other delimiter/escape would work better (example: newline for records, "^^" for fields, "^"-style character escaping; or JS-style "\"-escaping with field separator being "\N")

Reply on news.ycombinator.com