HNHacker News
TopNewBestAskShowJobs

jviotti

394 karma · joined November 5, 2014

https://www.jviotti.com
submissionscomments
jviotti··on Ask HN: What are you working on (September 2024)?
I'm working on JSON BinPack (https://jsonbinpack.sourcemeta.com), a binary serialization format for JSON (think of it as a Protobuf alternative) with a strong focus on space-efficiency for reducing costs when transferring structured data over 5G and satellite transceivers for robotics, IoT, automotive, etc.

If you work at any of those industries and pay a lot for data transfer, please reach out! I'm trying to talk to as many people as possible to make sure JSON BinPack fits their use case well (I'm trying to build a business around it).

It was originally designed during my (award-winning!) research at the University of Oxford (https://www.jviotti.com/dissertation.pdf), and it was proven to be more space-efficient than any tested alternative in every single tested case (https://arxiv.org/abs/2211.12799), beating Protocol Buffers for up to ~75%.

While designing it was already difficult, implementing a C++ production-ready version has proven to be very tricky, leading me to branch off to various other pre-requisite projects, like an ultra-fast JSON Schema compiler for nano-second schema validation (https://github.com/sourcemeta/jsontoolkit) (for which I'm publishing a paper soon).

jviotti··on How to succeed in MrBeast production (Leaked PDF)
I Googled a bit (I'm not a consumer of these types of YouTube videos) and apparently this guy is pretty rich by now. He identified a path to wealth that clearly works and exploited it well, so he is definitely smart. Well deserved from that point of view.

However, it makes me sad that we as a society even let what he does (a zero-net as many pointed out) be a path to wealth on the first place. The fact that we reward so much something that is so void of value is our fault and IMHO what needs to be corrected somehow. It's sad to think we throw so much money at people covered by ketchup, but many PhD's barely make any money and we cannot figure out how to make open-source sustainable.

If our reward schemes were more aligned with actually providing value to society, this guy might have applied his intelligence and amassed his wealth doing something more worthwhile for everybody.

jviotti··on Which open-source projects are widely used but maintained by just a few people?
JSON Schema is the industry standard schema language for JSON, used by big players like OpenAPI and by an insanely high number of APIs and products out there.

The TSC core team stays at ~6 people (I'm one of them)

jviotti··on Launch HN: Sorcerer (YC S24) – Weather balloons that collect more data
Compared to other serialisation formats, JSON BinPack analyses your data and derives custom encoding rules that are specific to the data at hand given all the context it had on it. That's why on JSON BinPack, the static analysis part is the most complex one by far, and why I'm building so much JSON Schema advanced tooling in https://github.com/sourcemeta/jsontoolkit (i.e. check the huge API surface for JSON Schema in the docs: https://jsontoolkit.sourcemeta.com/group__jsonschema.html)

Of course there is still a lot to do, but the idea being that what you get with JSON BinPack is extremely close to what you would have done for manually encoding your data, except that you don't have to worry about encoding things yourself :) Thus you get the best of both worlds: the nicety of JSON and the space-efficiency of manual encoding.

jviotti··on Launch HN: Sorcerer (YC S24) – Weather balloons that collect more data
Sounds like you are pretty familiar with satellite transmission at the hardware level. If so, I would love to chat to get your brains on it. I don't know much of the hardware constraints myself.
jviotti··on Launch HN: Sorcerer (YC S24) – Weather balloons that collect more data
Yeah, it all depends very much, given how huge the "embedded/IoT" spectrum is. Each use case has its own unique constraints, which makes it very hard to give general advice.
jviotti··on Launch HN: Sorcerer (YC S24) – Weather balloons that collect more data
It surprised me how popular this message got. I love nerding out about binary serialization and space-efficiency and great to see I'm not the only one :)

If you want to get deeper, I published two (publicly available) deep papers studying the current state of JSON-compatible binary serialization that you might enjoy. They study in a lot of detail technologies like Protocol Buffers, CBOR, MessagePack, and others that were mentioned in the thread:

- https://arxiv.org/abs/2201.02089

- https://arxiv.org/abs/2201.03051

Hope they are useful!

jviotti··on Launch HN: Sorcerer (YC S24) – Weather balloons that collect more data
Great seeing you over here Michael :) For other people reading this thread, Michael and I are collaborating on a paper covering the schema compiler I've been working on for JSON BinPack. Funny coincidence!
jviotti··on Launch HN: Sorcerer (YC S24) – Weather balloons that collect more data
Exactly! The real hardcore savings will always be when you pass a schema, as JSON BinPack uses that to derive smarter encoding rules.

However, schema-less is very useful too. The idea with supporting both is that clients can start with schema-less, without bothering with schemas, and already get some space-efficiency. But once they are using JSON BinPack, they can start incrementally adding schema information on the messages they care about, etc to start squeezing out more performance.

Compare that with i.e. Protocol Buffers, which pretty much forces you to use schemas from the beginning and it can be a bit of a barrier for some projects, mainly at the beginning.

jviotti··on Launch HN: Sorcerer (YC S24) – Weather balloons that collect more data
I never used cborium, but if I'm understanding it correctly, I think it adds types at the language deserialisation stage and not over the wire. Which means that it makes it a lot more ergonomic to use within Rust, but doesn't use typings for space-efficiency over the wire.
jviotti··on Launch HN: Sorcerer (YC S24) – Weather balloons that collect more data
We do! See https://benchmark.sourcemeta.com for a live benchmark and https://arxiv.org/abs/2211.12799 for a more detailed academic benchmark.

The TLDR is that is that if you use JSON BinPack on schema-less mode, its still more space-efficient than MessagePack but not by a huge margin (depends on the type of data of course). But if you start passing a JSON Schema along with your data, the results become way smaller.

Please reach out to jv@jviotti.com. I would love to discuss your use case more.

jviotti··on Launch HN: Sorcerer (YC S24) – Weather balloons that collect more data
Good question! Compared to Protocol Buffers and Apache Avro, that each have their own specialised schema languages created by them, for them, JSON BinPack taps into the popular and industry-standard JSON Schema language.

That means that you can use any tooling/approach from the wide JSON Schema ecosystem to manage schema evolution. A popular one from the decentralised systems world is Cambria (https://www.inkandswitch.com/cambria/).

That said, I do recognise that schema evolution tech in the JSON Schema world is not as great as it should be. I'm a TSC member of JSON Schema and a few of us are definitely thinking hard on this problem too and trying to make it even better that the competition.

jviotti··on Launch HN: Sorcerer (YC S24) – Weather balloons that collect more data
CBOR is a schema-less binary format. JSON BinPack supports both schema-less (like CBOR) and schema-driven (with JSON Schema) modes. Even on the schema-less mode, JSON BinPack is more space-efficient than CBOR. See https://benchmark.sourcemeta.com for a live benchmark and https://arxiv.org/abs/2211.12799 for a more detailed academic benchmark
jviotti··on Launch HN: Sorcerer (YC S24) – Weather balloons that collect more data
Fair point! "Embedded" and "IoT" are overloaded terms. For example, you find "IoT" devices all the way from extremely low powered micro-controllers to Linux-based ones with plenty of power and they are all considered "embedded". I'll take notes to improve the wording.

That said, the production-ready implementation of JSON BinPack is designed to run on low powered devices and still provide those same benefits.

A lot of the current work is happening at https://github.com/sourcemeta/jsontoolkit, a dependency of JSON BinPack that implements a state-of-the-art JSON Schema compiler (I'm a TSC member of JSON Schema btw) to do fast and efficient schema evaluation within JSON BinPack on low powered devices compared to the current prototype (which requires schema evaluation for resolving logical schema operators). Just an example of the complex runtime-efficiency tracks we are pursuing.

jviotti··on Launch HN: Sorcerer (YC S24) – Weather balloons that collect more data
That's the hardware. I meant on the software side through the transceiver. If you transfer less bits through the satellite transceiver, I believe you can probably reduce costs.
jviotti··on Launch HN: Sorcerer (YC S24) – Weather balloons that collect more data
Very cool! How are the balloons transferring telemetry back to earth for analysis, etc?

Asking because my research at the University of Oxford was around hyper space-efficient data transfer from remote locations for a fraction of the price.

The result was an award-winning technology (https://jsonbinpack.sourcemeta.com) to serialise plain JSON that was proven to be more space-efficient than every tested alternative (including Protocol Buffers, Apache Avro, ASN.1, etc) in every tested case (https://arxiv.org/abs/2211.12799).

If it's interesting, I'd love to connect and discuss (jv@jviotti.com) how at least the open-source offering could help.

jviotti··on Ask HN: Are people out there still using XML Schema?
> As a TSC member of JSON Schema, the best DX thing you can push through would be the xsi:schemaLocation replacement because right now mapping files to their schema is some damn tomfoolery

I'm not an XML Schema expert, and I might be missing something, but I believe what you are looking for is the "$schema" keyword (https://www.learnjsonschema.com/2020-12/core/schema/). You set it to a URI reference that can either point to a remote resource (i.e. over HTTP) or a local one using the "file://" scheme.

jviotti··on Why is Chile so long?
From the post: "You see how red Chilean Spanish is? It means it’s quite different. In the beginning of Grad School, I could understand all my Hispanic classmates, but I had a hard time understanding the Chileans!"

As a native Spanish speaker, I find this quite interesting. For me, Chileans are within the category of "easy to understand" while I might struggle a bit understanding some accents that the article qualifies as easy or normal.

jviotti··on Coup Attempt in Bolivia
I was in Bolivia during the event. It was not a real "coup" and it was in general a pretty uneventful military vs government dispute. Locals were a bit upset that the media did so much fuss about something that was not a big deal.
jviotti··on Demystifying the protobuf wire format
I'm currently building a Protocol Buffers alternative that uses JSON Schema instead: https://jsonbinpack.sourcemeta.com/. It was proven on research to be as or more space-efficient than any considered alternative (https://arxiv.org/abs/2211.12799).

However, it is still heavily under development and not ready for production use. Definitely looking for GitHub Sponsors or other type of funding to support it :)

jviotti··on Demystifying the protobuf wire format
I did a lot of research on binary serialization at the University of Oxford. One of the papers I published is a comprehensive review of existing JSON-compatible serialization formats (https://arxiv.org/abs/2201.02089). It touches on Protocol Buffers (and >10 other formats) and I'm analyzing the resulting hexadecimals close to how the OP is doing.

I also published a space-efficiency benchmark of those same formats (https://arxiv.org/abs/2201.03051) and ended up creating https://jsonbinpack.sourcemeta.com as a proposed technology that does binary serialization of JSON using JSON Schema.

jviotti··on Unikraft Launches KraftCloud: Never Pay for Idle Again
Awesome work! I've been playing around with Unikraft for a while now, support it natively on most of my C++ projects (at https://github.com/sourcemeta) and already deploying a bunch of serious things to KraftCloud. The whole experience is stunning. Very easy to use and VERY fast.
jviotti··on Ask HN: How much are you spending on SIM data transfer in IoT?
How many times are you recharging 500 MBs per year for the fleet? Also, what protocol and serialization technologies are you using the transmit the GPS data? For example, JSON, Protocol Buffers, etc?
jviotti··on Cloudflare employee posts layoff call with HR and goes viral [video]
This is ridiculous. More people should be following Brittany's approach and record these things. These HR shit shows are only going to end when people make them public and hurt the image of these companies. You did great Brittany.
jviotti··on Is Objective-C BOOL a boolean type? It depends
Author here! We write Objective-C++ to connect existing Objective-C code (that uses BOOL) to C++ (where we want to just expose `bool`), which got us into this fun rabbit hole :)
jviotti··on Writing books remains a tough way to make a living
I'm in the process of releasing my first book, with O'Reilly, this year (https://learning.oreilly.com/library/view/unifying-business-...). I didn't do it for the money (but for the recognition), but I'm very interested to know how it will economically fare.
jviotti··on Happy New Year HN!
Happy New Year everybody from Argentina, and may your wildest dreams come true!
jviotti··on Cold-blooded software
Great article. I try to follow this advice as much as I can. My personal website (https://www.jviotti.com) runs almost purely on well established UNIX tools like Make, Pandoc, Sed, etc. Repository here: https://github.com/jviotti/website.
jviotti··on Tell HN: Merry Christmas
Merry Christmas to everybody in this wonderful community. I sincerily hope all your dreams come true and that you have a bright and successful year ahead!
jviotti··on WebGPU now available for testing in Safari Technology Preview
Do you have examples? I use Safari as a daily driver, don't even have Google Chrome installed, and everything works great for me.
← PreviousPage 2 of 4Next →