Comment by jph | Hacker News Reader

Comment by jph | Hacker News Reader

Excellent, thank you for creating UXY!

I will donate $50 to you or your favorite charity to encourage a new feature: to-usv, which outputs Unicode separated values (USV) with unit separator U+241F and record separator U+241E.

Unicode separated values (USV) are much like comma separated values (CSV), tab separated values (TSV) a.k.a. tab delimited format (TDF), and ASCII separated values (ASV) a.k.a. DEL (Delimited ASCII).

The advantages of USV for me are that USV handles text that happens to contain commas and/or tabs and/or newlines, and also having a visual character representation.

For example USV is great for me within typical source code, such as Unix scripts, because the characters show up, and also easy to copy/paste, and also easy to use within various kinds of editor search boxes.

Bonus: if the programming implementation of to-usv calls a more-flexible function that takes a unit separator string and a record separator string, then you can easily create similar commands for to-asv, to-csv, etc.

Eventually you have to deal with content that contains your separator characters, however obscure. So essentially you have two choices:

A. use some "weird" separators and hope those don't appear in your input

B. bite the bullet and escape and parse properly

Option A is perfectly reasonable for one-offs, where you can handle exceptional cases or know they won't occur because you know what's in the data. However for reusable code, you need option B, which means not using `cut` to parse CSV files, for instance (since commas can occur inside double-quoted strings). In that case, what's the benefit of using USV over an existing, more common, format?

Yes you're exactly right about escaping.

Orthogonal to escaping, the choice is what characters to use for unit separator and record separator.

If the data are for machines only, then for me the choice of characters doesn't matter. If the data are potentially for reading or editing, such as by a programmer, then my choice is to prefer typically-visible characters over typically-invisible characters and/or zero-width characters (e.g. ASV a.k.a. DEL a.k.a. ASCII 30 & 31).

My choice of USV is thus because U+241F and U+241E are visible, and also in Unicode they are semantically meaningful.

Glad we agree on escaping.

I'm still not sure what the value is over CSV, which also has visible delimiters. It's true that you have to establish/enforce a specific convention around escaping and quoting, since CSV has historical variation here. But it would make more sense to me to encourage any particular consistent handling of CSV, rather than yet another entirely new separator. At least some tools already support CSV, whereas nothing currently supports USV, as far as I know.

U+241E is "SYMBOL FOR RECORD SEPARATOR". It seems a bit weird to use that as a separator instead of simply U+1E which is the ASCII character "record separator".

While not exactly what you asked for, I wrote something similar called csvquote ( https://github.com/dbro/csvquote ) which transforms "typical" CSV or TSV data to use the ASCII characters for field separators and record separators, and also allows for a reverse transform back to regular CSV or TSV files.

It is handy for pipelining UNIX commands so that they can handle data that includes commas and newlines inside fields. In this example, csvquote is used twice in the pipeline, first at the beginning to make the transformation to ASCII separators and then at the end to undo the transformation so that the separators are human-readable.

> csvquote foobar.csv | cut -d ',' -f 5 | sort | uniq -c | csvquote -u

It doesn't yet have any built-in awareness of UTF or multi-byte characters, but I'd be happy to receive a pull request if it's something you're able to offer.

How is USV better than ASV, which would use U+001E and U+001F?

Also, is your offer available for other tabular data tools? :)

USV is better that ASV for me because USV is visible.

For example I can write code samples such as:

  echo 'a␟b␟c␞d␟e␟f␞g␟h␟i' | tr ␟␞ '\t\n'
  a    b    c
  d    e    f
  g    h    i

Yes my offer is available for other tabular data tools. I want USV to become a good choice for data exchange. Message me at my contact information in my profile here.

I think you're going to need a bigger budget to establish your new proposed standard through consulting fees. Do you remember what happened to GCC's CHILL frontend?

Reply on news.ycombinator.com