There are similar problems with csv. A deceptively simple format with many variants to handle various edge cases.
Off the top of my head, you need to specify text encoding, line endings, a quoting mechanism in case a field contains the separator, and an escaping mechanism in case a field contains multiple lines or the quote character.
If you have those specified and everyone follows the specification, there shouldn't be any problems, right?
Woe be unto you if you have to process input data where those are not used consistently, of course.
Second, this: http://georgemauer.net/2017/10/07/csv-injection.html
And what I wrote before is basically all you need for a specification. Yes, it sucks that there is no fixed one that everyone uses.
Well, yes, but I feel that that's the point at which CSV definitely earns its "ill-specified format" hat.
Given that there is no way to extract the file specification from the file itself, you can only follow the specification if someone tells you the specification beforehand. You can even use the "wrong" specification and don't notice it because your specification overlaps with the "true" specification in the types of data you're typically seeing (looking at you, optional quotes).
You forgot the localized decimal separator symbol.
it never ends
Arguably right, practically useless.
All file formats have limits for how much they define types for the payload. Most have very limited types. JSON for example knows objects, arrays, strings, numbers, booleans and null. It's not that big a difference to have a format that has only strings.
You also need to specify the separator itself, which despite the name isn't always a comma, but can also be a semicolon or (less commonly) a tab.
https://www.iana.org/assignments/media-types/text/tab-separa...
They are there, they are convenient, and they don't require any visible character sacrifices.
'\0'
Depending on the source this might be:
- quote back-slash zero quote
- zero
- back-slash zero
- a byte with the numeric value of 0
- quote zero quote
- quote back-slash zero quote
- quote byte-with-value-0 quote
- malformed and should be ignored
Worse the same application might mix different of this quoting mechanisms for different columns or even in the same column. Normally it shouldn't but it's defintely a thing you can find in programs which don't use a proper marschalling library/code, and you know. CSV is easy so you surly don't need to add a library dependency just to use it. (sorry sarcasm).
Or with other words "because CSV is easy" it's somteimes not properly specified and sometimes no "proper" marschalling/serialization library/module is used resulting in ad-hoc fixes for quoting where needed which potentially don't follow any specification either.
Salespeople enter things like Company Foo, the One (which used to be Bar). The worst part is that often these are legal names, which means you do need to store them.
When I worked for a FAANG this was one of my major annoyances as there was no canonical number for a customer, so everyone did string matching which broke whenever a company changed their name (like for instance, if they'd just gone public).
Just say no to CSV's with commas in quotes.
It's almost like they really wanted to avoid this issue back in the 70s so it is built right into ASCII and unicode.
I wonder how you embed US/RS/GS/FS within a field though. Prefix with ESC?
As many as there are localized datetime formats :)
there shouldn't be any problems, right?
That would be reallys awesome, yes. With the aid of the other sibling comments you already figured by now it's not as simple, unfortunately.
- Values in most columns are escaped, but a few are not. (You have to count commas from both ends of the line and then log a warning and make a guess if there are too many commas.)
- ASCII whitespace is url-encoded (why?) but no other characters are.
- Columns without double quotes can only contain digits.
- Double commas are used as the field separator because they didn't want to implement quoting. (Some of the fields came from user input on a web page, and I prayed for them to hit a double comma and learn the error of their ways, but it never happened. I think their front end developers were sanitizing for them.)
- And my favorite, no quoting was supported at all, double quotes were left unquoted, commas were stripped from values when writing, and when reading, fields were compared against a list of mappings like "Acme Inc." -> "Acme, Inc." to restore the commas.
In fact, one of the biggest questions I've still yet to answer is whether Docker or OCI have ever clarified just which TAR format must be followed? As far as I can tell, it's a mix of "whatever library X supports"? https://www.cyphar.com/blog/post/20190121-ociv2-images-i-tar for example.
I'm not sure I want a more complicated format, but I do wish somebody would pick a new file extension for their next use of tar so we could clearly differentiate one tar format from another the next time someone builds on top of simple tar files for their binary distribution, etc.