Hacker News Reader

Comment by mro_name | Hacker News Reader

I recently learned via https://news.ycombinator.com/item?id=31257248 that ASCII has the idea of records and fields ever since. It's just not used, but workaround CSV.

No improvement of CSV handling will ever improved on that.

View on HN

Discussed here before, not everybody was seeing only positives.

"ASCII Delimited Text – Not CSV or TAB delimited text": https://news.ycombinator.com/item?id=7474600

A big problem is tooling - what they currently support, and the fact that excel will f*ck up anything.

I suppose the best way would be a suite of tools to edit them that are compatible with existing editors? TBH a great value-add for adoption would be versioning s.t. you can see (and revert) when other tools mess up your files.

See the "Conventions for lossless conversion to TSV" on the current version of https://en.wikipedia.org/wiki/Tab-separated_values . There are really only 3 chars to escape - the escape char, and 2 delimiters - to make everything easy to deal with (even binary data in fields - what "lossless" means here).

I'm just gonna leave this here.

https://git.sr.ht/~sforman/sv/tree/master/item/sv.py

(Python code for ASCII-separated values. TL;DR: it's so stupid simple that I feel any objection to using this format has just got to be wrong. It's like "Hey, we don't have to keep stabbing ourselves in the face." ... "But how will we use our knives?" See? It's like that.)

> any objection to using this format

How do I edit the data file in a text editor?

More generally, how do I edit the entire data file, including field/record/etc seperators, in a editor that can display only characters that are valid content of data fields?

(Not that CSV (barring de-facto-nonstandard extensions like rfc4180's `"b CRLF bb"` and `"b""bb"`) is any good for this either, of course.)

indeed, if the text editor doesn't care, it's no help. Same for all other characters - be it tab, cr, lf, crlf, ä, and so on.

Choose your tools wisely, I'm not aware the situation.

> > any objection to using this format

> How do I edit the data file in a text editor?

Use a non-sucky text editor.

For example, AFAICR NotePad++ displays those characters. Windows doesn't have any ultra-easy way to input them, but within NP++ you can copy-paste them.

> For example, AFAICR NotePad++ displays those characters.

Great! Now go back to the other half of the requirement: how do I store all characters that NotePad++ displays in the content of a data field?

These delimiters, you mean? You don't. They're delimiters, they never go in a field. Bam, problem solved. (Or rather, problem didn't exist in the first place.)

Could one get away with regular AWK using these combined with a streaming to/from CSV converter?

Not just "regular awk", but "regular most things", like head, tail, grep, etc. The "things" just need to be 8-bit/NUL handling clean and the data could even have binary fields. I mean, if there is a library like Python/Go/etc. that you especially trust to write a streaming converter then that is a fine approach, but otherwise see both https://news.ycombinator.com/item?id=31352517 and https://news.ycombinator.com/item?id=31352704 (at least!)

Yes and I suspect that this is why it was never added to AWK. I,and I am sure most people, have an AWK filter to transform their csv or whatever format to a format that AWK can use with an appropriate FS and RS.

Their "but why" section should really go into more detail about why filters are not 100% if your data has the possibility of containing your preferred line/record separator.

But real world, I have never had a situation where a csv-to-awk filter (written in awk of course) did not work.

Closely related- I remain mystified as to why FIX Protocol chose to use control characters to separate fields, but used SOH rather than RS/US or the like.

Because of EBCDIC.

Reply on news.ycombinator.com