Hacker News Reader

Comment by Beltiras | Hacker News Reader

This actually does. I used Python gzip to see how the minified would compress and came up with 157 bytes to JCOF's 134. Using the same mechanism to gzip the JCOF saves only 4 bytes. I'm actually rather impressed.

View on HN

It might on small files but it doesn't on (some) bigger ones. e.g. I dumped a Minecraft region file into JSON giving 85M. `jcof` turns this into 66M. `gzip` on the original json gives 14M for `-1` and 11M for `-9`. `zstd` also gives 11M.

But because the Minecraft JSON is highly redundant, I also tried it on 84M of fake social graph data. `jcof` gave 44M, `gzip -9` gave 19M and `zstd` gave 18M.

In summary, neat idea, but you're maybe better off with `gzip` or `zstd` in the real world.

JCOF and gzip are not mutex.

No, but when you get ~95% of the benefit (per other people's data) while keeping all the flexibility, compatibility with everything under the sun, and native-performance parsing everywhere, the supposed improvement looks a lot less like one.

A sibling has the data. For that case it's 10% of egress which can be a healthy chunk of change on your bill. For all such use cases you should of course benchmark a lot and then pick what fits. I think this tool is a nice one to have in the toolbox. The low-rate files seem to be close to optimal.

> For that case it's 10% of egress which can be a healthy chunk of change on your bill. For all such use cases you should of course benchmark a lot and then pick what fits.

Provided that you don't run into the issue of one end only talking in language X, which does not have JCOF support due to there not being any libraries for it yet. In the GitHub repo, it seems like there is only the JavaScript reference implementation for now.

The design documents are really thorough. I don't think I would be more than a day making a serializer in Python for example. Lack of libraries just means an interested developer needs to tend to the matter and solve the issue.

> A sibling has the data.

And now has updated data showing that sorting your keys lets `lzma` beat `jcof|lzma` at every level. The egress bill has been reduced! All rejoice!

EDIT: Nope, I was wrong, I was doing `lzma` against `jcof|zstd`. `jcof|lzma` is still sneaking ~1M below `lzma` at all levels.

You just blew my mind that “mutex” is a portmanteau of “mutually exclusive.” How have I been programming for 20 years and never realized this?

Perhaps you haven't had to program concurrencies or thought much about bitwise operators. It's not very frequent anymore. Used to be.

See https://news.ycombinator.com/item?id=32111239 where I tested gzip vs jcof vs jcof+gzip (tl;dr: you save 2M at the cost of an extra CPU intensive process)

I think people would reject your samples because the data size is too small. Production JSON comes in all shapes and sizes and situations where you care about performance usually indicates much larger payloads.

If your egress is paramount, you should of course benchmark everything to shreds. Then pick what fits.

Reply on news.ycombinator.com