Hacker News Reader

Comment by m0zg | Hacker News Reader

Not all protobufs. Protobuf 2 can do it just fine. The decision to not have emptiness support is the dumbest part of Proto 3 IMO.

View on HN

I don't think it's that crazy a decision, since it was also kind of crazy to pay the cost of "was this field set" for every single field, when the vast majority of the time you're not going to use it.

But it can require a bit more more thought in your protocol design. So for example in this case, the options would be

- design things such that 0 is ok to mean "most current" (so start revisions at 1) (this is hard after the fact, but if you know from day 0 that missing values for int types will be 0, you can design everything to start at 1) (Edit: maybe this is how etcd works?)

- explicitly break out revision into a message type (so you can notice if it's not provided)

- use something like "-1" to mean "now", so that 0 isn't overloaded

etc...

(You could argue maybe the right call for proto3 was to have a flag on a field saying if you want to be able to notice if it was provided or not. Best of all worlds, at cost of a bit of complexity.)

"Is this field set" costs 1 bit per _optional_ field in Proto2. _Required_ fields (which are also supported by proto2 and not supported by proto3) do not incur this cost. Those flags then would get packed into a bit mask. Not too big a cost, if you ask me.

But now you can't figure this out at all without adding another, boolean field, _and setting it separately_, which I'm pretty sure nobody is going to do unless they really have to, leading to the type of issue we're seeing here.

At my place of work, we typically follow the "message type" solution in situations like this. I don't think it's the most legible solution, but it's the best we can do with the proto spec: I always feel like I should qualify these fields with a comment explaining the apparently pointless wrapper.

Google themselves provide https://github.com/protocolbuffers/protobuf/blob/master/src/... to deal with this situation.

Quoted from the documentation:

> Wrappers for primitive (non-message) types.

> These types are useful for places where we need to distinguish between the absence of a primitive typed field and its default value.

It should probably be advertised more, as we've experienced that default values of optional fields are a surprising feature for smart developers who are new to protobufs. Maybe it's seen as a wart in the design? Getting rid of Null is hard.

I kinda wish they went with the approach that all fields are required, unless they are explicitly declared as optional. This is how Rust does it, and people seem to like it.

I think you got it backwards. The current situation where unset field results in a value is the equivalent of "null", it's the opposite of "getting rid of null". The previous design didn't have that problem.

I might be misunderstanding you, but the current situation results in a “zero” value, whereas “Null” would represent “unspecified”, or the absence of a value. I wouldn’t say that the current spec supports null, unless you’re using a wrapper like the one linked.

It’s the distinction between an optional type and an optional value (default value). With optional values but no optional types, you can’t be certain about the caller’s intentions. It’s a distinction that’s subtle but important, therefore a “gotcha”.

Getting rid of null is a noble idea, because of the headaches that null tends to induce. Optional types (like Rust’s) is a neat way to get the behavior of null without the value of null. Proto3 doesn’t have null, it’s replaced with arcane wrapping that’s arguably less straightforward.

Please let me know if I’m talking past you, it isn’t intentional, just late in the day. :)

The elegant way to do this is via coproducts and products.

It depends on the target language. I think the driving force behind the protobuf3 simplifications were to make working with Go better. Go pretty much required dropping the 'required' and non-zero-value defaults. Which annoys Python developers and similar with access to a None, nil or NULL value. It is particularly visible on my main code base, where I'm dealing with SQL with NULL, to a Go gRPC server which needs specially handle all those NULLable columns, via protobufv3 to Python clients which generally now can't tell if the original data was an empty string or NULL (because the solution to that is worse than the problem, which is annoying but manageable). protobuf3 is designed to be cross language, which in this case meant going to the lowest common denominator rather than making it work harder.

Yes, I would love if there were set/unset bits available. Even if it was awkward.

you’re probably right. obviously you can model it in go (just use a ptr!), but it doesn’t feel as natural.

You can implement emptiness with the wrapper message types, e.g. StringValue: https://github.com/protocolbuffers/protobuf/blob/master/src/...

and sometimes defining your own, e.g. nullable lists.

It's not elegant, but it is simple.

But the point is, proto2 had this feature, and it worked without any hacks.

Reply on news.ycombinator.com