A new Go API for Protocol Buffers
blog.golang.org
blog.golang.org
> The google.golang.org/protobuf module is APIv2. We have taken advantage of the need to change the import path to switch to one that is not tied to a specific hosting provider.
That's such a weird choice, and will be quite confusing. Now every time you see protobuf being imported, you have to be sure to pay extra attention to the domain used, and correctly remember which one is v1 or v2. If only go modules established a clear way to differentiate API versions using the import path!
It did, they just chose not to follow it :/ I wonder what rsc thinks about this decisions
However, the goal is to create a new package name that can be concurrently imported with previous versions. Everything else is just method. Don't confuse the goal with the method.
Are version numbers 2.0 and higher explicitly banned?
Edit: found an explanation further down (spoiler: not a very good one but a very go one).
Valid versions are in the 0 < x < 1.0 range.
> the version number alone should be enough to unambiguously differentiate between APIv1 and APIv2.
...which is how version numbers normally work. They've made a big breaking change in a 1.X release, so it isn't semver, but that isn't the end of the world. It works around having to put the major version number in the import path too, so really it works like most other non-Go, non-semver, projects.
>According to the Import Compatiability Rule for versioned Go, this implies that the new proto package must have a new import path. Since "github.com/protobuf/proto/v2" will incur great confusion with the proto2 syntax, we may take this time to use an import path like "google.golang.org/proto" (similar to how gRPC lives at "google.golang.org/grpc").
On the Rust side, macros can[1] transform .proto files into data structure definitions at compile time, and instances of the structs passed around have the benefit of strong typing, code completion in editors, lints and tests can easily validate properties of protobuf objects and so on. And all without adding more codegen steps which is nice - the macros can replace a build system having to do codegen as a separate step.
How often do you need to mutate any protocol buffer and remove (redact as they say) fields generating a completely invalid protocol buffer object? Will anything downstream match the ".proto" of an object that's missing a bunch of required fields? Wouldn't it be easier to just all of the sensitive data into an optional "sensitive_data" field that's defined by another .proto, and strip that?
[1] - https://github.com/danburkert/prost#generated-code-example
Whether a field is "sensitive" may change after the field is defined and used all over the place, so you can't just move it without changing all the code. But you can easily add a option (like [non_sensitive=true]) to any field so your redaction code picks it up.
A great usecase for this is if you want to store each row in a database as a proto message. Instantiate a descriptor based on the table schema and go.
Why not?
> A great example of this is if you want to use dynamically generated protocol buffers.
When would you want to do that? If your data isn't statically typed, isn't it more appropriate to use a type designed for dynamic shapes (e.g. a list or hashmap)?
> A great usecase for this is if you want to store each row in a database as a proto message.
Wouldn't a reasonable approach to this be to turn each row into a list (of variants) and serialize that?
Yes, it is, especially since the former can be used to implement the latter.
> Will anything downstream match the ".proto" of an object that's missing a bunch of required fields?
You may have missed it, but in proto3, required fields have been removed. All fields are optional now. https://stackoverflow.com/a/31814967/1102119 TL;DR: it's a code evolution problem.
In Go, a code generator converts the .proto files into Go code, which give the same benefits you see with the Rust macros. Its arguable if a code generation step is better or worse than macros.
The reflection is needed if you have received a protobuf the code don't have a definition for. You can now implement something like a gRPC proxy or gRPC load balancer in Go, without needing to compile it with code from the specific .proto files. You also appear to be able to access annotations on the message definitions, which are not embedded in the generated Go structs. Rust may well have similar features in its API. Java certainly does. A gRPC proxy is a use case redact sensitive data, for when you use it to create an audit log of the messages in the requests and responses.
Without the definition (and knowing the type!) there won't be anything around to tell the reflection API what the names, types and annotations are. All you would have is field numbers mapped to opaque blobs of data.
I don't need elaborate, brittle shell scripts to correctly update my compiler macros in order of dependency. I only need the compiler.
I'd rather just have generics.
Code generation for protos is nice for several reasons, but using it as an alternative to generics is not nice at all. I work on a number of projects with generics in internal classes that are not exposed as an API, and using code generation just to get those to work seems like a very big hammer for a small nail.
> The google.golang.org/protobuf module is APIv2.
> We have taken advantage of the need to change the import path to switch to one that is not tied to a specific hosting provider.
> (Why start at version v1.20.0? To provide clarity. We do not anticipate APIv1 to ever reach v1.20.0, so the version number alone should be enough to unambiguously differentiate between APIv1 and APIv2.)
Stuff like this is what makes Go confusing to newcomers.
For some time I couldn’t understand why people were complaining so much about —the now deprecated— GOPATH environment variable, and later the confusion of how to use all the different and incompatible dependency managers. Then, suddenly last year, it hits me… Go is confusing to people who haven’t followed the language from the beginning.
I started programming in Go in 2013 so for me it was very easy to adapt to every change in the language and the community.
I can only imagine all the ambiguous stuff people have to understand when learning the language in 2019-2020.
Having the build tools a core part of your language itself is both a strength and a weakness for Go, I think.
I've been writing Go since 2015ish and this is news to me...
However, it has become clear to me that it was a little too simple to handle all the new requirements and additions that came after v1.
That's how you get weird comments-as-code (// +build linux,386 darwin,!cgo), strings-as-attributes (tags), tacked-on modules versioning / vendoring, non-typed generics( interface{} everywhere), somewhat-typed generics (go generate), etc.
Go was not quite ready to handle all these directions people wanted the language to stretch to and as a result it's now already full of annoying warts.
I'll still pick Go if I need a simple service with amazing performance. Funnily enough though, I have no need for any of the (non-performance) additions that came after 1.6.
We could tag the new API v2:
+ Makes the "v1" and "v2" distinction very clear in the import path.
- Confusing: google.golang.org/protobuf@v1 doesn't exist, but v2 does.
- In ten years, hopefully nobody cares about the old github.com/golang/protobuf and the confusion is gone.
We could tag the new API v1:
- Less visually distinct in the import path.
+ Seems to make sense for the first version of google.golang.org/protobuf to be a v1.
+ If we decide it was a terrible idea, it's easier to go from v1 to v2 than to roll back from v2 to v1.
We waffled back and forth for quite a while on this, and eventually decided that the first version of google.golang.org/protobuf would be a v1. Then as we got closer to release (but with a certain amount of usage of v0 in the wild), we decided not to second-guess that decision but to start with a version that wouldn't overlap with any version of github.com/golang/protobuf to avoid confusion when someone reports a bug in "v1.0.1".
Maybe it was the wrong choice. If it was the worst choice we've made in the new API, I'll be happy!
I find this a super confusing and unintuitive choice, but I suppose it's too late for changing it now.
Sincerely, an othwerwise mostly happy protobuf user
Version it as google.golang.org@v2? google.golang.org@v3 (but @v2 doesn't exist)? Release an @v2 which is identical to @v1?
Rust .22 with breaking changes, will be breaking 4 years after that as well!
This seems like a good example why tying the name of the package in the code with the place where it can be downloaded was a very bad idea. Simply changing hosting providers is now a backwards compatibility break for your clients.
If Go imports didn't work this way, you could have simply offered the old v1 from both github.com and google.golang.org, and used a clear v2 for the v2. Sure , you would have similar problems if you wanted to change the logical package structure for some reason (say, when an open-source package moves to a different organization), but that is a much rarer case than switching code repos.
However, given that that ship has probably long sailed, you probably picked the right choice.
I would also note that v2 in the import path feels a bit strange, since it makes users of the newest code need to know about the history of the package, but there are also clear advantages which probably out-weigh this (I believe this is absolutely not the case for the decision to include the hosting provider in the import path).
See "go help importpath" for details on how this works.
> google.golang.org/protobuf is not tied to any particular provider, and we can redirect it wherever we want.
I agree, this is confusing. Why doesn't google.golang.org/protobuf@v1 return what's at github.com/golang/protobuf, then, if only to provide a tidy answer for those who wonder what v1 looked like?
I would strongly suggest using /v2 at the end. /v1 not existing on *.golang.org is not a problem in practice, but if it ends up one you could just set that up as a vanity import pointing to github.com for the old version.
[0] https://sagikazarmark.hu/blog/vanity-import-paths-in-go/
Also:
> In ten years, hopefully nobody cares about the old github.com/golang/protobuf and the confusion is gone.
is the corollary that this change will cause 10 years of confusion?
This would not be confusing at all. The UUID package I use just skipped from major version 3 to major version 7 to avoid confusion with UUIDv4, UUIDv5 and a possibe UUIDv6. It was well-documented, and even if it wasn't, the upgrade was absolutely painless, as it would have been either way.
e.g. github is now owned by Microsoft and it's a little awkward.
If Google won out on their attempt to buy Github, they wouldn't wouldn't be talking about a "specific hosting provider".
Though it's a little weird that people take this at face value, when the majority of the userbase was questioning this since the first release.
What if those things move to different domains, or different directory structures? Wouldn't that break every Go program that depends on them?
I've been using Java, Python, and Ruby for a while. They all have various pain points for managing dependencies but they do get one thing right: packages have names and versions, names are opaque strings that are not coupled to a particular hosting provider, and they don't care what directory on my laptop I use to store my dependencies and my code.
The Go dependency model seems like a real step backwards in usability and maintenance. It's like they are forcing one software org's standards on the rest of us. We don't all work at Google! Just let us depend on things that have names and version numbers!
Are there any plans to fix this for the whole language, for others as well? i.e. use namespaces, module names in the source but move the actual github.com, google.com or geocities.com URLs outside.
One thing which saddens me a lot though, is returning types you have to type assert because they had to avoid an import cycle. I don’t know if it could be avoided (probably not) but it’s still unergonomic.
> (Why the type assertion? Since the generated descriptorpb package depends on protoreflect, the protoreflect package can't return the concrete options type without causing an import cycle.)
One perennial annoyance is how awkwardly some Protobuf stuff ends up being represented in Go. For example, "oneof" types:
message Event {
oneof payload {
Create create = 1;
Delete delete = 2;
}
}
message Create {
string id = 1;
}
message Delete {
string version = 1;
}
type Event struct {
Payload isEvent_Payload `protobuf_oneof:"item"`
}
type isEvent_Payload interface {
isEvent_Payload()
MarshalTo([]byte) (int, error)
Size() int
}
type Event_Create struct {
Create *Create `protobuf:"bytes,1,opt,name=create,proto3,oneof"`
}
type Event_Delete struct {
Delete *Delete `protobuf:"bytes,1,opt,name=delete,proto3,oneof"`
}
Event{
Payload: &Event_Create{
Create: &Create{ID: "123"},
},
},
Event{
Payload: &Create{ID: "123"},
},
func nextEvent() isEvent_Payload { // <-- not possible
if something {
return &Create{...}
} else {
return &Delete{...}
}
}
There are other issues. Overall, the whole package seems designed from the bottom up for machines (mumbles possibly also by machines), not humans.The upshot is that using Protobuf types as "first-class" types — meaning the types you actually use internally in the meat of your app, as opposed to in the controller glue that lives in your API and mediates between the API and the internals — feels super messy.
As an aside, anyone know what issues this paragraph refers to?
The google.golang.org/protobuf/encoding/protojson package
converts protocol buffer messages to and from JSON using the
canonical JSON mapping, and fixes a number of issues with the
old jsonpb package that were difficult to change without
causing problems for existing users.
> The jsonpb package is intended to the be faithful implementation of the proto<->JSON specification, for which the C++ implementation is considered the canonical "reference" implementation.
And that C++ implementation does some kinda dumb things. For example, it marshals the protobuf type of int64 into a string when marshaling from a protobuf struct into JSON.
https://github.com/golang/protobuf/pull/916
I believe that the package they mention there is meant to be a more "canonical" protobuf <-> json marshaller than the existing jsonpb package.
Eventually they realized that oneof isnt an optional feature of proto - but the solution always felt rushed and bolted on because it was.
I hope the second try at proto api addresses these warts. I'm generally a golang fan - but not in this area.
Citations:
Maybe tagging fields with auth_userid and/or auth_token.
If you want to change the import path and also release a new version, then simply changing the import path is enough. No need to put v2.
See it as them releasing a totally new package that just happens to have the same name, if that helps.
For the average app? Overkill, sure.
I like PB, but it was too hard in my application to sync all of the sides with the same proto as it was developing rapidly.
For me schemaless was better, and now we have extremely easy JSON conversion... but did have to write the parser myself.
I personally can’t agree on PB being “the future”.
There's even GRPC-Web being worked on to bring it to the browser. That being said, it can be overused just like anything else.
I found this which shows json isn't too much worse than others, but it doesn't go into extreme cases