Comment by kstenerud | Hacker News Reader

Comment by kstenerud | Hacker News Reader

Parent Full thread

There are a number of problems with JSON and these schemas based upon it. And a huge chunk of the problems would be solved with a better, stronger type system. That's why I've spent the past couple of years building a replacement [1], [2], and an RPC layer and service selection mechanism that I haven't spec'd out yet.

We need a binary encoding because they're more efficient in code, time, and power. But we also need a text encoding because we're humans, and eyeballing binary data is terrible. I've made them both [3] [4], and they're 100% type compatible. The idea is that machines communicate using binary, and translate to/from text only when a human needs to inspect it or modify it.

The thing slowing me down atm is the coding side of things ([5] [6] [7] [8]) because it's tricky getting go to play ball with its reflect mechanism. Once I have a functioning gob replacement, I can move up to the next layer.

I'm probably a month away from releasing v1.0 of CBE and CTE, after which it'll be another few months for Streamux, and then the layers on top of that should go pretty quickly because I'm not bit bashing anymore.

[1] https://github.com/kstenerud/concise-encoding#concise-encodi...

[2] https://github.com/kstenerud/streamux/#streamux

[3] https://github.com/kstenerud/concise-encoding/blob/master/ct...

[4] https://github.com/kstenerud/concise-encoding/blob/master/cb...

[5] https://github.com/kstenerud/go-cbe

[6] https://github.com/kstenerud/go-cte

[7] https://github.com/kstenerud/go-compact-time

[8] https://github.com/kstenerud/go-compact-float

Great project, regarding the stricter type system we are currently working on a stricter JSON Schema version called TypeSchema which basically provides a strong type system following TypeScript. Through this you can use all tools which are currently working with JsonSchema but it is easily possible to generate code or build a binary representation. You can take a look at it here: https://typeschema.org/

Out of curiosity, why support references, like YAML?

References are necessary in order to support cyclic data. They're also useful for repetitive data.

How does your offering compare to capnproto?

It's similar in a number of ways. The communications protocols both are asynchronous, but capn'proto isn't inherently bidirectional (where commands can be sent both ways). Cap'n proto data is schema-based, whereas Concise encoding is ad-hoc.

Then again, Streamux is VERY low level, and I haven't written the higher level RPC mechanism yet (so it won't do a lot of things the Capn proto RPC layer does without some help). I'll be stealing a lot of high level ideas once I get to that level, but the primary thing is that Streamux is a communication layer designed to be built upon, since not everything requires the same heavy duty RPC treatment. Streamux will fit just fine in a tiny i2c environment or full on TCP or whatever comms channel you're using because you can tune it to use as much or as little of space/time/range resources as you need. And endpoints can negotiate these without even needing round-trips.

As for the binary format itself, Capn'proto relies upon schemas for decoding the data, whereas CBE encodes the basic data type so that it can be decoded without a schema (like JSON, XML, etc) so that you can encode ad-hoc data without needing to specify how it's decoded. Capn'proto only has void, bool, int (signed/unsigned), float, binary, list, and map types. Concise encoding has nil, bool, int (signed unsigned), float (binary and decimal), string, date, URI, binary, list, map, markup, comments, metadata, and references. References are important because they handle cyclic and repeating data in an elegant manner. Capn'proto doesn't support float compression.

That's off the top of my head. Capn'proto has a lot of good ideas and looks very nice for its niche, but the biggest issues I have are the need for a schema, lack of types, and the lack of a twin text format (and this is by design, because it's competing with Protobufs, not JSON). However, it supports mmap, so it can be blindingly fast for certain applications that concise encoding won't even try to compete in.

I just realized that I haven't added Capn'proto to my features matrix: https://github.com/kstenerud/concise-encoding#comparison-to-...

Reply on news.ycombinator.com