Comment by lifthrasiir | Hacker News Reader

Comment by lifthrasiir | Hacker News Reader

Parent Full thread

Oh, hello! My gut feeling was that csv-next is pointing to the informal specification (and they read like it, but specific to a single problem point).

I guess you might not have understood my complaint (my bad), so let me give some examples how the specification should look like.

An informal specification looks like this [1]. You should give examples and rules enough to use your format and sufficient to implement most of the things. You should give definitions for keywords that are specific to your specification (but you can omit common definitions). You should give some (but probably not all) ideas where the specification can go wrong: Unicode whitespaces, Byte Order Mark, platform-specific newlines, escape sequences, duplicate keys in the front matter (well, actually there is no provision how to put the front matter at all), numeric overflows, and so on.

A formal specification looks like this [2]. In addition to what an informal specification provides, you should give lots of examples and formalized rules (most frequently ABNF [3]) to implement all the things. There should be clear and reasonable error handling policies. The wording of the specification should be clear, unambiguous and preferably standardized (there are specific meanings to "MUST", "SHOULD" etc. [4]). The specification should be honest about its pros and cons. You should be explicit about the flexibility of the format: you should give a list of what can be extended or modified later and what can't.

I strongly suggest you to put an informal specification at the least, and to prepare for the eventual development of a formal specification by pondering about missing pieces (it is a daunting task, I know).

[1] https://github.com/toml-lang/toml/blob/bb47759841ac368d86eb7...

[2] https://tools.ietf.org/html/rfc7049

[3] https://en.wikipedia.org/wiki/Augmented_Backus%E2%80%93Naur_...

[4] https://tools.ietf.org/html/bcp14

I think you don't understand :-). A specification doesn't always have to be complicated, that's the point. I'm all for tech notes with all the details. Here's a challenge for you - write a csv parser - I know - it's a daunting task. For inspiration, here's an example from your humble self - https://github.com/csv11/csvreader

> A specification doesn't always have to be complicated, that's the point.

That's right, but your "specification" was not even enough for writing a CSV 1.1 file. For example I have mentioned a front matter issue---there is no example using it, and I'm deeply confused how the front matter looks like (or even where it is). I'm seeing numeric units there, but I don't know how they are handled (or, say, what happens if different units like m² are there) from your page.

I'm not saying I'm a good specification writer (and I'm not even a native speaker of English), but I at least try. I have once written an informal specification [1] which should be almost enough to write valid files and start implementing parsers, without being too complicated. You can avoid a complicated specification without being too vague.

> For inspiration, here's an example from your humble self - https://github.com/csv11/csvreader

That's frankly much better (in terms of explicitness) than the front page, why not linking to it? :-)

But I believe that it is still not sufficient. Rather, I'm now unsure about the goal of your project: is it a set of Ruby libraries with fancy, modern but incompatible (by itself) CSV extensions? Or is it hopefully going to be a universally used format? If your intention was the former, then your front page should have been clear about it---it is not about a format but about a library. And if you are going to have a format, then my suggestions hold.

[1] https://github.com/lifthrasiir/cson (with a bit of formal materials, just for the clarity)

The goal of the project is to improve CSV :-) - it's in the title e.g. CSV Evolved and in the version 1.1 (that is, it's not version 2.0). It's about not breaking things - it's about small changes (mostly) for humble humans (not machines).

It's NOT about one universal csv format and the ultimate specification to settle the matter until the end of history etc.

Thanks for the detailed suggestions. I see your points. I appreciate your helpfulness.

A specification has to be, um, specified. For example, the first thing to explain is whether the file format is actually text and if it is what the allowed characters and encodings are. And let's not forget the big one: how are records and fields delimited? Too boring to explain?

On the whole, some ideas to make CSV files human-friendly, but neglecting backward compatibility (comments, use of spaces...) and introducing major syntactic and semantic cans of worms (multiline values, named fields, defaults...). I think CSV files should evolve towards tighter constraints instead.

You should stop using smilies in this passive-aggressive manner. You should also stop condescending to people. You may feel that you can't be wrong so they must not understand because they are too dumb and daunted by it, but you should not be revealing that to them in your comments, because it's so offputting.

Reply on news.ycombinator.com