Hacker News Reader

Comment by burntsushi | Hacker News Reader

We have that already, RFC 4180: https://tools.ietf.org/html/rfc4180

The problem is adoption. Most CSV parsers accept a superset of RFC 4180, because the only thing worse than not adhering to a good spec is tripping over CSV data in the wild.

View on HN

RFC 4180 does not specify the character encoding.

This isn't a problem with JSON, XML, ASN.1, Protobufs, or Thrift. Anything other than CSV, really.

Indeed it does, although not directly. From the RFC:

> TEXTDATA = %x20-21 / %x23-2B / %x2D-7E

So, it is a subset of ASCII whose hex codes must lie in the ranges given above. Basically, any printable ASCII character with the exception of 0x22 (= double quote) and 0x2C (=comma) which serve other purposes.

From "3. MIME Type Registration of text/csv":

  Common usage of CSV is US-ASCII, but other character sets defined
  by IANA for the "text" tree may be used in conjunction with the
  "charset" parameter.

Basically, you can specify that it's UTF-8 or whatever via the Content-Type response header. Setting this response header isn't something an encoder could do. A CSV parsing library also shouldn't be responsible for loading the CSV file via HTTP. In most cases, CSV files aren't directly fetched via HTTP anyways.

Anyhow, ASCII is utter nonsense. We aren't living in an ASCII world. Even the Americans don't do that. An ASCII-only format is completely useless.

Reply on news.ycombinator.com