Related example is the recently discussed problems with filenames, and the many pitfalls that scripts can stumble upon when handling them.
Related example is the recently discussed problems with filenames, and the many pitfalls that scripts can stumble upon when handling them.
The problem is that edge cases are not really edge cases… in certain locales. ;) Someone could be using commas as the decimal separator since that’s how they do it ’round these parts.[1][2] So it’s not like they’re storing “weird” stuff in tabular format, like HTML blog articles; they’re just storing numbers.
Of course, Excel and other software has tried to be way too clever about it with different separators depending on locale, which makes the whole pipeline complex and context-dependent.
[1] A surprising amount of countries have this convention, including my own… doesn’t make sense to me since it becomes harder to list numbers.
[2] And the number one selling point of CSV is that it is “human editable”, so of course someone is going to do that at some point in the pipeline.
The thing is that from the point of view of each individual tool, there isn't even an error! You ask awk to split on commas and give you the second field, it splits on commas and gives you the second field. It opened the file no problem, it didn't run out of memory writing its output to stdout... There were no errors to speak of from what it could see. The "error" is an error in the overall logic of the code.
<https://github.com/dbro/csvquote>
Using it with the first example command from this article would be
csvquote file.csv | awk -F, '{print $1}' | csvquote -u
By using the "-u" flag in the last step of the pipeline, all of the problematic quoted delimiters get restored.