It seems clear enough that protobufs were optimized in a scale that involved a lot of time/cost-sensitive reads and far fewer time/cost-sensitive writes. That's a valid tradeoff for Google scale, that comes at a cost of other potential applications (such as telemetry that can be very time/cost-sensitive at write-time but has much more relaxed read time/cost-sensitivity).
> Sure, having to recursively precompute lengths before serialization is a bit of a hassle, but I wouldn’t call it expensive.
It's expensive in memory (needing to prebuild all the subcomponents of the message, instead of streaming them as needed), and potentially time (prebuilding then sending, versus streaming as soon as data is available to write). Of course, in the scheme of things it may not be expensive to particular use cases. (Which seems why the author seems to particularly highlight this specific use case where such things really are expensive overhead necessary to avoid.)
Nah, you're assuming too much. Protobuf was thrown together in a fairly ad hoc way by a couple (brilliant!) engineers (Jeff and Sanjay) to help make the Google search index protocol easier to maintain. The specific design decisions in Protobuf were not carefully tested or weighed against other possibilities. They just did something that worked well enough, and it worked well enough that it was rapidly adopted by the rest of the company. It was then too late to change anything.
Yes, the fact that a variable-width size must be written before the data is a kind of big problem, which essentially requires you to make two passes over the message tree, one to compute sizes and one to write the data. There are some clever optimizations that can reduce the impact, but I don't think the designers would make the same decision if starting from scratch with no need to support legacy. But there was no point in history where it was worth breaking compatibility to fix this issue, so that's how it remains. It's a problem, just not that big a problem.
(I maintained Protobuf for several years, including writing version 2 and open sourcing it.)
Groups can be encoded in one pass, they are efficient to decode (unless you were trying to skip the sub-message, a la LazyField), and they don't have the string/message ambiguity in UnknownFieldSet that messages have.
Do you remember how that came to pass? Maybe some of this happened during the evolution of proto1, which was before my time.
I argued for switching to group encoding for submessages when working on proto2, but was shot down. It was a long time ago, but I think the counter-argument was some combination of "it's not worth the breakage" and "the ability to lazily parse sub-messages is too valuable".
And just to be clear, I don't think this is bad. On the contrary, I think Protobuf won because it did a wide variety of things "pretty well" while moving quickly and solving real problems. This is how the best technologies are usually made, not by academically trying to perfect everything, but by banging out something that works and running with it to solve real problems. If you try to carefully design everything perfectly upfront, you'll spend a huge amount of time on decisions that don't really matter.
I think this is a really important point. I don't think I've ever told you this before, but I am really impressed by how quickly you turned out proto2. While there are some things here and there that we wish we could change, a lot of it holds up really well. A lot of decisions I made in upb early on, where I thought I was improving on proto2, actually turned out to be bad ideas and the proto2 design was better.
Though in retrospect, I don't know if proto2 was a net win, given all the migration pain it caused. If I were doing it again I would take a more incremental improvement approach on proto1. It would have taken a lot longer but with less pain, I think. That said... who knows if that would have resulted in a better or worse outcome. Open sourcing would have taken a lot longer.
Had I realized at the time that I was taking on a project with no good solutions... heh.
There are lot of improvements we can make without touching API, but whenever the API itself is a barrier to further improvements, there are just few good options for managing that.
Your point about incremental changes reminds me of the Linus rant about "bundling", where he argues that incremental changes lead to a better result than a big bang rewrite: https://yarchive.net/comp/linux/bundling.html I very much agree with that approach when possible, but Linux has the benefit of a much narrower API offered to its users. The protobuf API is not only the API of the core library, but of every generated class. The massive surface area just makes any kind of change to generated APIs an enormous challenge (for example, returning string_view from accessors instead of std::string).
That's fair enough as a description of how the protocol was (not) designed that those sorts of trade-offs were not taken into account in the initial design.
However, it doesn't entirely invalidate my assumption: search index protocol is pretty obviously a context/scale that would have deeply favored more time/cost-sensitive reads (need to get search results to users as fast as possible) and the time/cost for writers less of a pressure (as writers for the index protocol could presumably be amortized with caching/proxying/throwing hardware at the problem). Whether that was an intentional "optimization" process or simply "optimization evolutionary pressure", it does seem to me (as an entirely outside observer) like a natural trade-off optimization that ocurred at Google scale for protobufs that kept protobufs feeling "well enough" that they were essentially left alone and never optimized for something with different trade-offs (such as use cases that were more write than read-heavy).
Which it is still useful to know those sorts of "optimization evolutionary pressures" to answer questions like "sure, it worked for Google, but will it work for me in this very different use case/scale?"
I didn’t like it until I was forced to use it. Now I’m not sure I’ll ever go back.
There’s a handy code snippet to turn any protobuf message into JSON:
from google.protobuf.json_format import MessageToJson
import json
from pprint import pprint as pp
pp(json.loads(MessageToJson(msg)))
Examples of turning arbitrary tensorflow objects into JSON: https://twitter.com/theshawwn/status/1331625739993575427?s=2...(As a response to a sibling of the parent, the same rules apply to internal code, CapWords should only be used for consistency within a file. All new code should be embrace it's snakey heritage).
[0]: https://google.github.io/styleguide/pyguide.html#3162-naming...