Hacker News Reader

Comment by tptacek | Hacker News Reader

Performance is a real issue for Go's standard JSON, but this is a lot of extra boilerplate code to have to write (I'd probably codegen most of this if I had to), so I'd assume the reasonable strategy would be to implement with encoding/json, profile, and then just GoJay the hotspots.

View on HN

It looks like this has a Marshal and Unmarshal function just like it's core library counterpart. So I'd guess you might be able to use this as a drop in replacement. However I'm yet to prove that theory.

encoding/json's Marshal and Unmarshall uses reflect and struct tags. This library doesn't: you have to define a function to make your struct satisfy an interface.

Macros would solve this problem nicely if they we're available. I sure wish first class macros were on the golang roadmap.

If you want to code in a language with macros, pick a language with macros. They tend not to look like Go (or C, or C++), because part of what makes macros effective is that coding in those languages is pretty close to working with their ASTs. Writing a Lisp-like macro for Go sounds like it would actually be more annoying than simply writing a codegen for it.

I don't think there's anything precluding Go from making it possible, other than the maintainers' desire to not have macros. Rust gives you the power here, and as a result has some tremendous serialization libs:

https://github.com/serde-rs/serde

Macros don't need to be as simple to use as in lisps to be useful - the rare library like serialization that can really benefit from them can be worth the language feature

Rust is a good counterexample, yep.

The end result with macros is far more seamless than with codegen, the point being that a consumer of the library doesn't even need to know about the macros. The code just works like the reflection approach, but without the runtime penalty. Codegen on the other hand makes an implementation detail a maintenance cost for the user of the library.

The end result with macros is far more seamless than with codegen, the point being that a consumer of the library doesn't even need to know about the macros.

A double edged sword, that! Sometimes the end result with things like macros, is that it so seamless, that there is precious little to figure out what went wrong. Often, if your nifty new facility seems magic, its bugs are going to seem doubly magic. Not too long ago, I had to debug an exception for which no source code existed. It only existed as the confluence of 3 C++ templates.

Codegen on the other hand makes an implementation detail a maintenance cost for the user of the library.

I've heard veteran programmers say that there should be a separation of labor when it comes to libraries. Only certain people make good library writers, in the same way that only certain people make excellent musical accompanists. To be an excellent accompanist, the accompanist should have some sense of what it's like to be the lead. (So should themselves be able to take the lead.) The accompanist should have the perspective to not let his ego get in the way. The accompanist should be giving the lead what she wants, but modulated through their own expertise and sense of taste. (As opposed to mindlessly always giving everything asked for.)

If there is a separation of responsibility, why would codegen be such a burden? If the library has to change so often, perhaps the responsibilities aren't distributed optimally?

Macros usually make debugging easier than code generators do. That's because quasiquote and similar operators let you debug code, not code that generates code.

I'm the first to admit that macros have downsides compared to not having them in the language, but ease of debugging generated code isn't one of those downsides.

I've generally found that C/C++ macros make debugging harder. Debuggers don't always deal with debugging them the most straightforward thing.

That's because quasiquote and similar operators let you debug code, not code that generates code.

What I've found is that you can debug the generated code to debug the code generator. Code generators shouldn't be used to do something terribly complicated. I'd agree with you that code generators doing complicated things is a code smell. Leave those to native syntax, provided it's not as badly thought out as templates.

> C/C++ macros

I'd argue that those aren't first class macros. C macros are just source manipulators, and C++ templates (I'm not super experienced with them so forgive come if I'm way off) don't really manipulate the AST, it's more a type system with compile time resolution.

> A double edged sword, that! Sometimes the end result with things like macros, is that it so seamless, that there is precious little to figure out what went wrong.

First class macros generally contain the same logic as a reflection based approach, but they execute at compile-time and memoize the result. If you can debug compilation (which languages with first class macros generally support), then debugging is roughly the same. There are definitely cases where that's not true, but in this domain (serialization) a macro based solution is well understood. See https://github.com/devinus/poison/blob/62e98f19552289f3f7139... for a macro based example in Elixir.

S-expressions aren't actually necessary for macros to work. You just have to be able to separate "READ" from "EXPAND" (to use Lisp-ish terminology).

I believe, but am not sure, that this is possible for Go. (Of course, whether this is a good idea is another thing entirely. I probably wouldn't have included macros if I had been put in charge of Go 1.0, given Go's goals of simplicity.)

More on this from my former colleague Dave Herman, who did his dissertation on macros: http://calculist.org/blog/2012/04/17/homoiconicity-isnt-the-...

An alternative to both Lisp-like macros and code generation is to trigger macros via annotations in the syntax, and for the macro to manipulate the AST directly using the "go/ast" package. Apache Groovy, which has the same curly-braces style syntax as Golang, does it like this.

You don't have to use structs tags (nor even structs) to use encoding/json. In fact I often don't as a lot of my usage is with maps rather than structs.

I did some testing with gojay and it did work as a drop in replacement for most of my usage. However the performance improvements I saw in my benchmarks were not nearly as favourable as those published in the projects git repository. I'm sure I could get better results if I played around with their APIs a little more rather than just using the mashallers but frankly the utility I'm using it in favours the flexibility of encoding/json a little more anyway, even if that does cost me a little in performance (and to be clear, it really wasn't much as I needed to iterate the execution of my utility a thousand times just to get any meaningful differences between the two libraries. So we're not talking real world usages).

That said, if you're building high performance servers then I'm sure gojay would really stand out. On reflection (no pun intended), my requirements wasn't really the target use case for this package.

That's fine, but this library isn't a drop-in replacement for encoding/json; to be that, it has to work for people who expect tagged structs to round-trip through it, which won't happen with this library.

That's fine but I never stated "drop in for all use cases". A point I made very clear in my second post. But for some use cases it can be. As I had explained already.

The rest should be abundantly clear which use case would apply to anyone who's spent more than five minutes in Go (or any programming language that supports reflection) and had read the first like of the packages readme (ie that it doesn't use reflection).

It's definitely worth remembering that one doesn't have to use structs and tags to write nor read JSON in Go before people start bitching about boiler plate code and lack of macros.

You mention codegening, which is common in Go for a variety of reasons (as I understand the language does not support so much as a red-black binary tree - or any other containers that aren't part of the syntax - without codegening).

An additional reason for codegening is the extreme syntactic simplicity of the language: there is little syntactic sugar and it has a tiny, simple, explicit specification.

On the other hand the syntax is also meant to be clear, unambiguous, and human-readable. I was thinking what is good for humans to read might not be the easiest to get programs to output.

Question: since you seem to know about Codegening, do you find Go's grammar, syntax, etc to be a good target for codegening?

I don't have a more specific question but you could talk about any other feelings you have about codegening against Go (or any other language)

Having written a data serialization library based on codegen, I would have to say that Go is relatively simple to codegen. One big annoyance is the import system of "if you don't need it, it's an error to import it.", which, while simple enough for manual code, means you have to do extra work while generating code to keep track of what you actually use (i.e. if none of the things are a time.Time, you don't need to import "time"). The other approach is to use stuff like "var _ = time.Now" kind of constructs, which do make the codegen easier, but I personally feel are far dirtier.

One very nice feature of go is that the formatter is available as a library (https://golang.org/pkg/go/format/#Source), so you can gen "raw" code and then throw it through the formatter to get nice pretty output.

Unused imports are definitely not an unsolved problem: as last step after dumbly generating code, run goimports.

Never said "unsolved", I said "annoyance". And I tend to write the code to figure it out and do the right thing. My annoyance is with code gens that do https://github.com/golang/protobuf/blob/master/protoc-gen-go.... Not even goimports will strip those...

The next milestone is actually to write a code generator for structs, maps and slices.

Out of curiosity, is their any consideration to run-time gen and compilation followed by dynamic loading? There would obviously be UX to startup time tradeoff involved..

I'm not sure how most languages tackle high performance json encoding/decoding, but I have seen a few .Net implementations that take advantage of the JIT to gain performance while providing a more seamless library consumer experience.

Dynamic loading for go is not well supported, and that kind of "on the fly" compilation, JIT or otherwise, is not supported at all...

That's exactly what go-codec does

Reply on news.ycombinator.com