Hacker News Reader

Comment by otabdeveloper4 | Hacker News Reader

All this effort on building a better JSON parser for language 'X' should have been better spent on inventing a statically-typed JSON. (Type inference seems like a thing that should work well in this space.)

View on HN

> All this effort on building a better JSON parser for language 'X' should have been better spent on inventing a statically-typed JSON

how does that help when you want to implement a JSON-based protocol ?

I'd guess that 99.9% of those protocols are actually statically-typed.

E.g., an object's "foobar" field is always an array of int, and won't suddenly become a string from one invocation to the next.

It seems incredibly strange that we're somehow not leveraging this.

What you're saying exists. Protobuf, flatbuffers, capnproto, xdr, parquet, arrow, avro, orc, sqlite. Indeed if you use these formats you will be able to load the data into memory more quickly than using JSON.

But this article isn't about those protocols. It's about JSON.

You can do it with JSON too. Various statically-typed languages do JSON parsing straight into structs all the time. (e.g. in the Rust ecosystem, Serde is the most popular implementation of this concept.)

Because it's relatively easy to make JSON-serialization fast if you know the shape ahead-of-time. It's a solved problem.

Here's a pure-JS solution: https://github.com/fastify/fast-json-stringify

> E.g., an object's "foobar" field is always an array of int, and won't suddenly become a string from one invocation to the next.

If only… I had to figure out how to deserialize a JSON API where a field could be: a) an object X, b) an array of X, or c) a string (mapping to X.Y)

That doesn’t need to be a problem. In Rust, the most popular thing in this space is Serde, with the serde_json crate. It can cope with such things just fine; define a data type that is capable of representing all the possibilities (perhaps as simple as an enum with a variant for each style, or perhaps it normalises them into one), and then… well, you probably have to write a visitor to help deserialise, which isn’t trivial, especially when you’re used to just deriving those things and so you’re not so familiar with Serde internals. But it’s possible. https://serde.rs/string-or-struct.html gives one example. The length of the example may be daunting, but it’s actually not too bad.

It was C#. I ended up writing a custom converter that converted all 3 cases into X[]

Hasn't that been done? Efforts like Amazon's Ion come to mind, or Google's Protocol Buffers & FlatBuffers.

http://amzn.github.io/ion-docs/

https://developers.google.com/protocol-buffers

https://google.github.io/flatbuffers/

ProtoBuf requires a schema for the document to be known ahead of time and involves deserialization; Ion is more like JSON in that it's self-describing, but has a rich set of data types as well as annotations on fields to further describe meaning. FlatBuffers requires a schema and provides access to the without parsing/unpacking.

On the other hand, if you're building a service and specifying its interface, rather than just specifying a data format, then there are tools like Smithy, gRPC, and Thrift.

https://github.com/awslabs/smithy

https://grpc.io/

https://thrift.apache.org/

> a statically-typed JSON

Do you mean "JSON with a schema" (there are multiple such) or "JSON decoding to static types" (without reflection).

The latter.

JSON is popular because it's schemaless; storing and versioning schemas is its own nightmare of inconsistent and inefficient hacks.

However, in practice, JSON messages almost always have an implied schema. Figuring it out on the fly shouldn't be too hard - just apply the JIT and type inference technologies that compilers have already used for decades.

(Not a problem with an immediately obvious solution, but that's why we're here, no?)

> However, in practice, JSON messages almost always have an implied schema.

Typed json already exists in various attempts eg tjson

Really I think the problem is that JSON is great when your schema is still in flux -- once it's public/stable, it makes sense to want to strictly encode the current schema.

The solution then is that you really want a way to trivially transition from "implied schema" to "strict schema" -- though a typed json won't help you too much because the larger trouble is figuring out what your schema is in practice. I guess you really want either a code analysis tool to determine it, or more likely an automated tool to take eg a list of json responses (probably based on a test suite/code coverage) and produce a typed json schema that you can simply drop into your public API

True, miss it also sometimes. It will not be as lightweight though - reminds me about SOAP/WSDL times.

GraphQL goes in this direction by enforcing type checks, but is designed for web frontends...

Reply on news.ycombinator.com