JSON is one of the best things to ever happen to software development.
It's a format that still bears excessive decoration (what's the purpose of quotes around field names? what are all those commas for?) yet it's limited in the types of data structures that it's able to express (natively). I'm not particularly fond of Clojure specifically but a format like EDN would have been superior in just about every way.
The datastructure complexity being limited is also a pretty significant key to its success. More complex datatypes means greater chances for JSON handling libraries to lack compatibility.
The only substantial shortcomings of JSON I see are shortcomings associated with any textual serialization format. Optimizing for human readability in a use case that's 99.99% of the time not read by a human.
That's literally how low you have to go.
"The only substantial shortcomings of JSON I see are shortcomings associated with any textual serialization format. Optimizing for human readability in a use case that's not 99.99% of the time not read by a human."
There are a few other shortcomings that are reasonably substantial/significant, but yeah, that's the gist of the problem.
JSON looks good compared to what we'd be using instead of JSON, which is nothing so nice and structured as XML. The competition to JSON is something infinitely more ad-hoc, probably without a distinct parser, such that a generic library to generate or consume it is impossible, and getting usable error messages is equally impossible.
> "The only substantial shortcomings of JSON I see are shortcomings associated with any textual serialization format. Optimizing for human readability in a use case that's not 99.99% of the time not read by a human."
I agree with this and disagree at the same time: Optimizing for human readability means optimizing for the weird case, the 0.01% (but it seems to be more often than that) of the time you need to go beyond the tools you have to fix something. Saying that's rare is true but inapt: Seatbelts are only used in rare cases, too.
It's like, "hey, we came up with a simple way to do it, but to make it easier to deal with this little edge condition that creates complexity, let's significantly up the complexity and number of edge conditions so they're endemic to the space, and then we're all good go".
The irony is, I invariably end up needing to use a computer to help me read JSON anyway.
If you've got good built in tooling for payload visualization, then you might have minimal overhead to debug from a text like format. Both protobufs and flatbuffers (not to mention BSON), have good tools that spit out JSON equivalents.
In reality however, there are some cases where protobufs, flatbuffers, or BSON are superior to JSON, but there are a lot of cases where they aren't. You'll have to weigh the pros and cons for each situation. And a lot of the time, there's not time to benchmark everything, so you kind of have to guess how the elements of the system are going to interact.
The one element that every system has is humans, so it's a fairly safe bet that humans will have to read whatever format you use.
I spend probably 15-45 minutes a day just in Postman, testing JSON calls. If something goes wrong, I'm inspecting requests/responses in Chrome. When we integrate a new team member, we don't have to have them install any tools--they're included in the browser they have installed. We don't have to write any schemas. When I start a new project, I don't have to install any libraries: they're included in my language(s). When we integrate with a partner company, we hand them sample requests/responses as text.
How many JSON documents per day do we have to send to get the payoffs you're claiming?
And my company is not unique: in fact, the stack I'm using is one of the most common stacks on the market.
In reality, now, JSON opens in everything from Chrome network inspector to Vim, and protobuffers/flatbuffers don't.
Chrome also decompresses gzip and understands TCP/IP, but it doesn't handle LZMA or SS7.
Sure, thanks to our cult-like following of bad principles, we've made support of on a pretty broken stack with lots of terrible consequences ubiquitous. I'd argue that's bug, not a feature.
I make a solid income solving problems that people pay me to solve. Why should I abandon that and devote my life to achieving 9% size and 4% availability time increase[1] that no client has ever asked me for?
And to be clear, it's not that I don't care about performance. It's that if my automated tests notice an endpoint loading slowly or if a client complains about performance, I can almost always achieve order-of-magnitude performance gains by optimizing a SQL query or twiddling some cache variables, which almost never happens by switching serialization formats. I have used protobuffers in a few cases, where profiling indicated it as a solution, but this has not been the norm in my experience.
The first optimization is getting it to work, and the second optimization is whatever profiling tells you it is.
[1] https://auth0.com/blog/beating-json-performance-with-protobu...
But I think you're right that as long as we stay this course there's going to be more problems and so you'll be able to make more money solving problems that didn't need to exist.
You're absolutely right about the first optimization being to get it work. You're just discounting the reality is that you're making it far more difficult for that to happen.
You can make tools that present data in any format in a way that is easy for humans to digest. Letting a very small and trivial part of the problem drive what is a much larger problem space is pretty flawed.
{"time":"2020-07-22T10:59:14.95406-04:00","message":"{\"level\":\"debug\",\"module\":\"system\",\"time\":\"2020-07-22T10:59:14.953909-04:00\",\"message\":\"Running MetricCollector.Flush()\"}"}
this is a very moderate example of what I deal with daily, all because JSON includes quotes around fields.
I wonder if it would’ve been better for them to Base64 encode their message. Of course, this itself presents other problems.